Google

Social may be the key to innovation as competition in search heats up

As reported around this time last year Yahoo and Microsoft have signed a $700 million deal which meant that Bing would provide Yahoo’s search results leaving our friends in Sunnyvale to run what will effectively be a content based web portal, one far more popular in the US than here or the rest of Europe. Clearly, this is all part of Microsoft’s offensive against Google, which has also included taking a stake in Facebook, thus leading a conglomerate of brands against Larry Page and Sergey Brin’s search giant. But now the competitive scramble for users in the search space seems to involve almost every trendy brand in digital.

However, regarding the specific Yahoo/Bing deal, things just started to get a little more real. Last week an update was sent to advertisers stating that Yahoo would being serving natural search results from Bing from “August or September onwards”. Moreover Yahoo will integrate its PPC ads to Microsoft’s AdCenter by the beginning of the ‘holiday season’ (that’s Christmas to us limeys) but may delay that until 2011 if it decides that would “improve the overall experience” for both advertisers and users. “If organic search results are an important source of referrals to your website, you’ll want to make sure that you’re prepared for this change,” so the email said. Well sure, 80% of internet journeys start with search and these two new found friends are important to the search market, though Google is still leading by far, more so in the UK than most places.

According to ComScore’s latest figures from last month, Google have 91.7% of the UK search market with Bing and Yahoo on 2.98% and 2.55% respectively, figures largely unchanged from the last quarter. In the US it’s a different ball game with Google on “just” 63.70%, Yahoo on 18.30% and Bing on 12.10%, with slight rises from the last two against Google over the last quarter.

So many hope that this deal will have a positive effect on search in terms of innovation. For a start, Google will have to try harder, especially in the States, something which will have a knock on affect to the rest of the world. The biggest reason for this is that the merger obviously means increased market share to around a third for Bing/Yahoo. Such an enlarged competitor means more advertisers who may have previously only used Google may experiment with AdCenter, meaning that Google will have to try harder to keep users using their brand, something they have managed quite well in the past from free applications such as Maps and Gmail, to paid for models like the mobile operating system Android and even a rumoured hardware rival to Apple’s iPad.

As SEO industry guru Danny Sullivan said last year, “If Microsoft can adopt a passion for innovation and push the envelope, Google will have to respond in kind. The search experience will evolve more rapidly, hopefully kicked out of the revenue obsessed stasis that it’s currently in. Stagnation benefits no one except the analysts and bean counters who insist that quarter over quarter performance is the only metric that matters. We’re way too early in the game to be that cautious and boring.”

In what form might this innovation come? Well, social could be the key to that. For over a year now it has been speculated that Google use more than PageRank to determine the rankings of web pages. Many search analysts believe that inbuilt into the algorithm are signals from offline media and social networks, even those, such as Twitter and Facebook, that have their links set to ‘nofollow’ (so no link equity is passed on). These links would not carry as much weight as a “regular link” but evidence has been recorded of increased natural search ranking even when no links have been involved. Most famous of these is the recent Magners example from eConsultancy.

Personally, I think it’s fair to say that nothing is certain at this stage, so little is with Google’s algorithm, but there is definitely more emphasis being put on social activity, mainly because since October last year Twitter’s main revenue stream has come from sharing data with Google and Bing, a process that began when Tweets started to show up in natural search results as the engines clambered over themselves to show more ‘real time’ information to the user.

Also, as blogged about by my colleague Johnny Gedye, location based social networking site Foursquare are in talks with Google and Microsoft for a similar deal to Twitter’s:

‘Speaking to the Telegraph, [Foursquare co-founder] Crowley said Foursquare was discussing partnerships with “everyone” – which would include search kings Google, Microsoft and Yahoo! – to “enrich” their search engines with trends generated by the location-based data.

“We can anonymise data and use it to show venues which are trending at that moment,” Crowley explained, voicing the example of Twitter, “Twitter helped the world and the search engines know what people are talking about,” he continued. “Foursquare would allow people to search for the types of place people are going to – and where is trending – not what.”’

And this isn’t the only area where location based networks are springing up. Last month Twitter itself launched Twitter Places whereby users are able to tag tweets to specific places (such as venues) and clicking on those location names will bring up recent tweets from those places. Whether this will become part of the data fed to Google and Microsoft remains to be seen but there is certainly a scramble to make location an integral part of the search experience. Facebook is also rumoured to be developing a similar offering, not to mention anything that may be being thrashed out with Gowalla.

No one knows who will come out on top of this but one thing is for sure, search is only going to become a richer channel over the next year and it looks likely that the brands that make best use of the social space will be the ones that benefit the most.

Tags: , , , , , , ,

1 comments Add This

Weekly Social Media Update

Twitter Promotes Deals with Earlybird

Twitter has created a promotional account to highlight time sensitive exclusive deals. @Earlybird will tweet about special offers, sneak-peeks, and events. At the moment deals are largely focused on US and international brands, but there is clearly potential to set up regional or interest-specific accounts. Building on Promoted Tweets and Promoted Trending Topics, Earlybird could turn into a very lucrative activity for Twitter.

Goodbye Facebook Gifts

Facebook Gifts will no longer be available from the 1st of August. Given that I don’t know anyone who still uses them, I think it’s probably fair to say that Facebook has outgrown this particular function, especially as apps allowing more sophisticated social gaming have emerged. A statement from Facebook focuses on the role gifts have played in the network’s development: “Out of the Gift Shop’s “gift credits” came the virtual currency, Facebook Credits, that now makes it easier for people to buy premium items across the many games and applications on Facebook.” So, credits are here to stay, but gifts are gone for good.

Springwise Innovation Insanity

Take a look at the 2010 Springwise list of business innovations: Innovation Insanity. Highlights include: QR codes for bikes, tube refunds via iPhone app, and crowdsourced outfit decisions.

Real Life Social Networks

Paul Adams, Senior User Experience Researcher at Google, has put together a great deck on how to better map real world social networks online. Real people don’t just have one big bucket of “friends”: they have complex overlapping webs of relationship groups. Therefore, online social networks must be designed for multiple independent social groups, and must respect the need for privacy at the intersection of those groups.

The Human Avatar

A slightly more sinister version of David On Demand, The Human Avatar is a promotional stunt for the game All Points Bulletin. One man has been chosen, and we, the public, can customize him according to our preferences. At the end of the project, a virtual version will be released onto the streets of the game in-world. Creepy.

New Graffiti Artwork from Blu

This is wonderful, and you should all watch it. Blu is an Italian graffiti artist best known for his stop-motion mural animations. His most recent work “Big Bang Big Boom” has been spreading across Twitter like wildfire.

Tags: , , , , , , , , , , , , , , ,

0 comments Share

Google and Facebook gear up to fight for social search

Recently it seems that Google can’t make enough enemies – once their primary target may have been Microsoft but if Google’s attitude to Apple is anything to go by Redmond’s lot seem positively irrelevant these days. And if the rumours surrounding Google Me are anything to go by it sounds like Facebook just made the top of the hate-list.

Me is allegedly Google’s attempt to move on the ‘full service’ social network space that is Facebook (yes, I did just coin a social media description) but despite rumours proclaiming this as a major deal it is difficult not to be just a little bit cynical.  We have already seen Google launch both Wave and Buzz to ridiculous hype rapidly followed by almost laughable silence weeks after their respective launches – why should Me be any different? And more importantly, why is Google not focusing on joining up all of their various social hooks into something that makes sense? At present they have a variety of different social offerings yet most of them act like the others don’t exist – from Google Voice and Chat through to Buzz, Wave and even Google Reader (with its built in sharing settings) the graph may usually move between them but little else does.

So maybe that is all Google Me really is – a platform to pull together all of Google’s other platforms. Yet it is already being labelled a competitor to Facebook – this despite the fact that having to rebuild a whole new social graph on a new social network is about as enjoyable as actually being forced to converse with most of those forgotten school friends you could passively ignore before the days of Facebook.

So why bother? Well it probably has something to do with the fact that Facebook overtook Google in the US earlier this year to become the biggest site in terms of visits and it shows no sign of slowing. Whilst Google’s core offering (adverts served against search results) doesn’t currently directly compete with Facebook’s (adverts served against personal content) Google have to be more than just a little bit conscious that it wouldn’t take much for Facebook to make a move into their space.

What makes Google great? They have vast amounts of data about sites, the relationships between sites and the ways in which people access those sites. And what do Facebook have? Vast amounts of data about people, the relationships between people and, since the introduction of the ‘like’ button, the ways in which people access and share sites.

Facebook have recently started including sites with ‘like’ functionality into the search results a user receives when they search for anything on the Facebook site. But to be brutally honest, it’s horrible – there is no relevance to the results and it doesn’t fit with the user behaviour for people on the site. Yet it isn’t inconceivable that Facebook could buy a search engine and if you began to lay social graph data combined with content consumption habits you could have the next evolution of search: results that are socially aware. Imagine a result page where the sites your friends visit frequently get a little boost in the results for your searches.

The social search engine has seemed an obvious next step for years and yet still hasn’t happened – probably partly because no single company has had the relevant data sets, they have typically sat in separate businesses.  Of course let’s not forget that privacy concerns are likely to be a huge factor too, since search is just so personal. Yet packaged in the right way, whereby both sharing and privacy controls are simple and straightforward, it becomes a tantalising prospect.

Facebook have said search isn’t their focus (but they would, wouldn’t they) yet Google’s continued focus on building relationship data certainly suggests that social search may be the future.

Ultimately Google Me will still crash and burn if it can’t offering something unique that Facebook doesn’t – trying to move a population of 400 million to a new home is no mean feat – but if Google spies a threat to their core search business then you can be sure they are about to throw everything they can at the social space.

This story was originally posted at The Wall.

Tags: , , , ,

0 comments Share

Bing to launch updated, renamed web crawler “Bing Bot”

Microsoft is to launch its new spider later this year. Here’s what site owners need to know.

Microsoft’s search engine wasn’t always called “Bing” and its web crawler, “msnbot”, hasn’t kept up with the name change. When Microsoft renamed Live Search (formerly MSN Search) Bing, we have to admit to being mildly disappointed that it didn’t take the opportunity to rename its spider “Bing Bot”.

There are many good reasons not to change the name of a spider, especially one as widely used as Microsoft’s search spider. Many software packages look at the name of visiting browsers and spiders (known as the User-Agent) to perform a variety of functions, and it’s possible that problems might occur for a time on less well-configured websites if this were to be changed. For example, Yahoo! maintained the User-Agent “Slurp” for its spider, which it inherited from its acquisition of Inktomi, to “ensure consistency and minimal disruption”.

It appears that Microsoft has decided that the branding “Bing Bot” is too good to miss, however, and has announced that its next generation spider will indeed be renamed when it comes out of beta.

Here’s what site owners need to know:

When is this happening?

This will happen on 1st October 2010.

This is also when Microsoft’s new spider will officially come out of beta.

What will the User-Agent be?

Microsoft’s current User-Agent is:

msnbot/2.0b (+http://search.msn.com/msnbot.htm)

The new Bing Bot User-Agent will be:

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

In addition to the “bingbot” branding, there are two other changes to note. Firstly, Microsoft is switching to the “Mozilla/5.0”-style User-Agent. Google made this change more than six years ago because it wanted web servers to treat its spider more like a real web browser. The second, more minor, change is that the “b” (meaning “beta”) in its version number has been dropped.

Any other changes to the spider’s requests?

In addition to the User-Agent change, Microsoft has also change the “From:” HTTP header field, so the old value of:

From: msnbot(at)microsoft.com

will become:

From: bingbot(at)microsoft.com

Will my old robots.txt entries still work?

Thankfully, Microsoft has decided to make its spider respect the User-Agent field which it currently recognises in robots.txt, “msnbot”. However, the way in which it will work from October is somewhat subtle, so deserves a brief explanation.

Whilst existing directives will still work, Microsoft is also going to recognise a “User-Agent:” robots.txt entry of “bingbot”, and it will give precedence to an entry of “bingbot” over an entry of “msnbot” (which, in turn, has precedence over the catch-all User-Agent entry of “*”). This means that, if you add robots.txt rules for “bingbot”, it will ignore all other rules, including those for “msnbot”.

Whilst adding conflicting “msnbot” and “bingbot” entries hopefully isn’t too likely to happen on most sites, in a larger, more complex organisation in which many different people or departments are able to make changes to robots.txt files, I wouldn’t be surprised to see someone accidentally trip up and add a new “bingbot” entry which doesn’t match up with the already existing “msnbot” entry (for example, where a separate “crawl-delay” value for Bing is specified).

Microsoft clearly wants site owners to update their robot.txt files with the new User-Agent, and we’d definitely recommend that you do this – but don’t forget that the new Bing Bot only launches on 1st October – until then, you should still use the old “msnbot” terminology in your robots.txt files.

What should I do now?

Firstly, if you currently have a separate robots.txt entry for msnbot on your site(s), make a note on your calendar on to change it to “bingbot” on October 1st.

Secondly, make sure that your website doesn’t do anything else special for Microsoft’s crawler or for visitors which don’t identify themselves as ‘Mozilla compatible’. This could include tools such as analytics packages or software which performs anti-spam functionality such as request rate-limiting.

Other than that, there shouldn’t be anything to worry about! However, in the (hopefully unlikely) event that you do experience any problems come October, Microsoft has set up an email address (bingbot@microsoft.com) to help to resolve any issues.

Tags: , , , , , , , ,

0 comments Share

Google Caffeine live.

Back in August we blogged about the news, from Google, of an update to its architecture.  Since then there has been much speculation in the industry about whether or not it was already live. Yesterday Google announced the official launch of its “Caffeine” update.
In Google’s own words

“Caffeine provides 50 percent fresher results for web searches than our last index, and it’s the largest collection of web content we’ve offered.”

Google’s head of spam also explained the update at an SMX advanced session captured on video for Search Engine Land. Matt’s key points in summary were:
Caffeine…

  • Instead of crawling millions of documents in one day and then pushing it live hours later – with the caffeine update  Google can crawl documents and immediately put them into the index to be served live seconds later. So the entire index becomes closer to real time.
  • Increases Google’s ability to scale up the capacity of its index (In the official Google blog post it says that Caffeine already uses nearly 100 million gigabytes of storage!)
  • Makes it easier for Google to annotate documents with information.

As this is an update to Google’s infrastructure, it should not affect rankings.

Tags: , , , ,

0 comments Share