LBi London

Google and Facebook gear up to fight for social search

Posted by Adam Russell July 2nd, 2010

Recently it seems that Google can’t make enough enemies – once their primary target may have been Microsoft but if Google’s attitude to Apple is anything to go by Redmond’s lot seem positively irrelevant these days. And if the rumours surrounding Google Me are anything to go by it sounds like Facebook just made the top of the hate-list.

Me is allegedly Google’s attempt to move on the ‘full service’ social network space that is Facebook (yes, I did just coin a social media description) but despite rumours proclaiming this as a major deal it is difficult not to be just a little bit cynical.  We have already seen Google launch both Wave and Buzz to ridiculous hype rapidly followed by almost laughable silence weeks after their respective launches – why should Me be any different? And more importantly, why is Google not focusing on joining up all of their various social hooks into something that makes sense? At present they have a variety of different social offerings yet most of them act like the others don’t exist – from Google Voice and Chat through to Buzz, Wave and even Google Reader (with its built in sharing settings) the graph may usually move between them but little else does.

So maybe that is all Google Me really is – a platform to pull together all of Google’s other platforms. Yet it is already being labelled a competitor to Facebook – this despite the fact that having to rebuild a whole new social graph on a new social network is about as enjoyable as actually being forced to converse with most of those forgotten school friends you could passively ignore before the days of Facebook.

So why bother? Well it probably has something to do with the fact that Facebook overtook Google in the US earlier this year to become the biggest site in terms of visits and it shows no sign of slowing. Whilst Google’s core offering (adverts served against search results) doesn’t currently directly compete with Facebook’s (adverts served against personal content) Google have to be more than just a little bit conscious that it wouldn’t take much for Facebook to make a move into their space.

What makes Google great? They have vast amounts of data about sites, the relationships between sites and the ways in which people access those sites. And what do Facebook have? Vast amounts of data about people, the relationships between people and, since the introduction of the ‘like’ button, the ways in which people access and share sites.

Facebook have recently started including sites with ‘like’ functionality into the search results a user receives when they search for anything on the Facebook site. But to be brutally honest, it’s horrible – there is no relevance to the results and it doesn’t fit with the user behaviour for people on the site. Yet it isn’t inconceivable that Facebook could buy a search engine and if you began to lay social graph data combined with content consumption habits you could have the next evolution of search: results that are socially aware. Imagine a result page where the sites your friends visit frequently get a little boost in the results for your searches.

The social search engine has seemed an obvious next step for years and yet still hasn’t happened – probably partly because no single company has had the relevant data sets, they have typically sat in separate businesses.  Of course let’s not forget that privacy concerns are likely to be a huge factor too, since search is just so personal. Yet packaged in the right way, whereby both sharing and privacy controls are simple and straightforward, it becomes a tantalising prospect.

Facebook have said search isn’t their focus (but they would, wouldn’t they) yet Google’s continued focus on building relationship data certainly suggests that social search may be the future.

Ultimately Google Me will still crash and burn if it can’t offering something unique that Facebook doesn’t – trying to move a population of 400 million to a new home is no mean feat – but if Google spies a threat to their core search business then you can be sure they are about to throw everything they can at the social space.

This story was originally posted at The Wall.

Tags: Facebook, Google, personalised search, search, social search

0 comments Share

Bing to launch updated, renamed web crawler “Bing Bot”

Posted by Ian Macfarlane June 30th, 2010

Microsoft is to launch its new spider later this year. Here’s what site owners need to know.

Microsoft’s search engine wasn’t always called “Bing” and its web crawler, “msnbot”, hasn’t kept up with the name change. When Microsoft renamed Live Search (formerly MSN Search) Bing, we have to admit to being mildly disappointed that it didn’t take the opportunity to rename its spider “Bing Bot”.

There are many good reasons not to change the name of a spider, especially one as widely used as Microsoft’s search spider. Many software packages look at the name of visiting browsers and spiders (known as the User-Agent) to perform a variety of functions, and it’s possible that problems might occur for a time on less well-configured websites if this were to be changed. For example, Yahoo! maintained the User-Agent “Slurp” for its spider, which it inherited from its acquisition of Inktomi, to “ensure consistency and minimal disruption”.

It appears that Microsoft has decided that the branding “Bing Bot” is too good to miss, however, and has announced that its next generation spider will indeed be renamed when it comes out of beta.

Here’s what site owners need to know:

When is this happening?

This will happen on 1^st October 2010.

This is also when Microsoft’s new spider will officially come out of beta.

What will the User-Agent be?

Microsoft’s current User-Agent is:

msnbot/2.0b (+http://search.msn.com/msnbot.htm)

The new Bing Bot User-Agent will be:

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

In addition to the “bingbot” branding, there are two other changes to note. Firstly, Microsoft is switching to the “Mozilla/5.0”-style User-Agent. Google made this change more than six years ago because it wanted web servers to treat its spider more like a real web browser. The second, more minor, change is that the “b” (meaning “beta”) in its version number has been dropped.

Any other changes to the spider’s requests?

In addition to the User-Agent change, Microsoft has also change the “From:” HTTP header field, so the old value of:

From: msnbot(at)microsoft.com

will become:

From: bingbot(at)microsoft.com

Will my old robots.txt entries still work?

Thankfully, Microsoft has decided to make its spider respect the User-Agent field which it currently recognises in robots.txt, “msnbot”. However, the way in which it will work from October is somewhat subtle, so deserves a brief explanation.

Whilst existing directives will still work, Microsoft is also going to recognise a “User-Agent:” robots.txt entry of “bingbot”, and it will give precedence to an entry of “bingbot” over an entry of “msnbot” (which, in turn, has precedence over the catch-all User-Agent entry of “*”). This means that, if you add robots.txt rules for “bingbot”, it will ignore all other rules, including those for “msnbot”.

Whilst adding conflicting “msnbot” and “bingbot” entries hopefully isn’t too likely to happen on most sites, in a larger, more complex organisation in which many different people or departments are able to make changes to robots.txt files, I wouldn’t be surprised to see someone accidentally trip up and add a new “bingbot” entry which doesn’t match up with the already existing “msnbot” entry (for example, where a separate “crawl-delay” value for Bing is specified).

Microsoft clearly wants site owners to update their robot.txt files with the new User-Agent, and we’d definitely recommend that you do this – but don’t forget that the new Bing Bot only launches on 1^st October – until then, you should still use the old “msnbot” terminology in your robots.txt files.

What should I do now?

Firstly, if you currently have a separate robots.txt entry for msnbot on your site(s), make a note on your calendar on to change it to “bingbot” on October 1^st.

Secondly, make sure that your website doesn’t do anything else special for Microsoft’s crawler or for visitors which don’t identify themselves as ‘Mozilla compatible’. This could include tools such as analytics packages or software which performs anti-spam functionality such as request rate-limiting.

Other than that, there shouldn’t be anything to worry about! However, in the (hopefully unlikely) event that you do experience any problems come October, Microsoft has set up an email address (bingbot@microsoft.com) to help to resolve any issues.

Tags: Bing, Google, Microsoft, Robots Exclusion Protocol, search, search engine, spiders, technical, yahoo!

0 comments Share

Google Caffeine live.

Posted by Rik Stewart June 9th, 2010

Back in August we blogged about the news, from Google, of an update to its architecture.  Since then there has been much speculation in the industry about whether or not it was already live. Yesterday Google announced the official launch of its “Caffeine” update.
In Google’s own words

“Caffeine provides 50 percent fresher results for web searches than our last index, and it’s the largest collection of web content we’ve offered.”

Google’s head of spam also explained the update at an SMX advanced session captured on video for Search Engine Land. Matt’s key points in summary were:
Caffeine…

• Instead of crawling millions of documents in one day and then pushing it live hours later – with the caffeine update  Google can crawl documents and immediately put them into the index to be served live seconds later. So the entire index becomes closer to real time.

• Increases Google’s ability to scale up the capacity of its index (In the official Google blog post it says that Caffeine already uses nearly 100 million gigabytes of storage!)

• Makes it easier for Google to annotate documents with information.

As this is an update to Google’s infrastructure, it should not affect rankings.

Tags: caffeine, Google, search, search engine updates, SEO

0 comments Share

Video distribution

Posted by Ian Macfarlane June 3rd, 2010

Now that you know how to optimise your videos for search it’s time to distribute them across the web.

Following from last week’s post about video SEO, this week’s post covers how to distribute your videos across the web and track their performance.

Whilst you can host video files on your own site and submit them to video search engines (Google, Yahoo!, Bing, Blinkx etc.), YouTube is the 800 lb gorilla in the video space. The only way in which your video will be found by people searching on the YouTube website is if you upload the video to YouTube itself.

Using YouTube also has additional benefits – YouTube automatically creates Media RSS feeds, which you can use to submit the video to search engines, and it also makes hosting of videos effectively “free”. However, the Media RSS feeds that YouTube provides link back to the YouTube page, not to the page on your site.

Therefore, we recommend hosting your videos on your own site as well as on YouTube. You can then generate your own Media RSS feeds (linking back to your site) and submit these feeds, rather than the YouTube feeds, to the various video search engines. If hosting videos on your own site, it may also be useful to provide the video content in multiple formats – the more formats in which the video is available, the larger the potential audience (although more formats also means additional costs in both time and bandwidth, so there is a definite trade-off involved).

It may be worth uploading the video to other video hosting platforms, such as Dailymotion, MetaCafe or Vimeo, as well as to YouTube.

When uploading your video to video hosting sites like these (as opposed to submitting it to video search engines), we recommend watermarking the video with the brand name to prevent it from being re-used without attribution.

Most video hosting sites allow you to include a URL along with each video – each video that you upload to a third-party site should ideally link back to the page on your site on which the video is hosted.

It is also a good idea to embed the URL of the page on your site where the video file is included within the video itself. Short URLs are generally better, as users will have to manually type them in. URL shortening services which support tracking of users are particular useful here: they can allow you to track users who visit your site after watching one of your videos, and identify which of your videos and which video hosting sites are attracting the most visitors.

Tags: best practice, Google, search, SEO, video

0 comments Share