If you have ever heard the business idiom that “Turnover is vanity, but profit is sanity”, in the same vein, it has long been said within the SEO community that “PageRank is vanity, but conversion is sanity”.

As accountants look at turnover for one sign of a healthy business, an SEO looks at PageRank as one factor of a healthy site; Along with many other more important factors, such as building a site which search engines can read, optimising for conversions and generating profit.

In SEO it can be easy to fall into the trap of analysis paralysis, spending too much time on figures and not enough time working out issues with your site and promoting your brand. Concentrating on creating a site which users love, search spiders can access, developing your product or service to be the best it can be, creating PR for your site and striving for good quality, natural links can all be achieved without the need to check any PageRank values.

In fact, as fellow SEO Peter Handley points out it’s arguable whether it would be better if Google switched it off altogether:

Add to this that Google is known to show Toolbar PageRank for sites which have been penalised and can’t even pass any “real” PageRank, that new pages (in the last 8 months now) don’t show a PageRank value and can continue to accrue PageRank over the life of the page, that PageRank passed per single link is diminished by the number of links on a page, and that links on certain areas of pages such as footers and sidebars are known to pass less value than others, it becomes very difficult to establish just how effective a link will be from PageRank alone.

While there is no official statement, there are also rumours that Google may switch off the Toolbar PageRank in the future, as they believe it fuels “manipulative” and ill-advised link building practices.

Whether they do or not, we will continue to consider Toolbar PageRank values for the time being, but as a fleeting measurement with a fistful of salt.

“Google Instant” is now live for visitors to Google.com who are in the U.S. and is also live for users who are logged into a Google Account in a few other countries, including the UK. Search results appear as a user types, updating in real time to match the query based on each additional letter. As the results change, the search suggestions recommended by Google are also updated, channelling users towards a set of suggested keywords. Together, these features could have a significant impact on advertisers, who will see a shift in user behaviour as a result of users becoming accustomed to this new way of searching.

For now, this will have no effect on Google search partner sites, mobile searches or browser toolbar searches for the moment, which make up a reasonable amount of searches. Chitika, a web analytics company estimates that 9% of all searches are completed through the default Firefox toolbar and custom Google homepage, which is not currently affected. Google has said that mobile searches will have Google Instant in the future, and a successful rollout would probably mean that more searches will be included in Google Instant going forward.

A major change in the way an ‘impression’ is counted will have some impact on figures reported by Google Webmaster Tools and Google AdWords. Traditionally, an impression is incurred each time a user clicks on a suggested search option or presses return. With Google Instant, impressions are also accrued when a user pauses on a set of results for three or more seconds, or clicks on a result as they appear while a user types.

In considering how Google Instant will play out, it is worth noting that Google doesn’t always get it right. When Google recently promoted their homepage backgrounds, such was the underestimation of the dislike for the new ‘feature’ that Google was not prepared for the backlash of disgruntled users. Not only this, but other products and services have failed to deliver, notably in the social arena where ‘Google Wave’ and ‘Google Buzz’ were not well received by the online community. So it may be that this radical change may not be so well received, and may not be a permanent feature.

Of course, there is a way to disable ‘instant’ search, by using the prominent drop down menu to the right of the main search box, and it remains to be seen how many users will disable this feature.

How to switch Google Instant off

Below is an image of the drop down box to the right of the main search box, select ‘Off (Press Enter to Search)’ to disable Google Instant.

How will this impact your SEO campaign?

Also, if you have a rounded SEO strategy working in tune with a believable brand, this may not be a bad thing at all, as higher quality results should surface due to a higher accuracy and a more relevant search experience. However, as results change strategies will need to be reviewed to ensure meeting up to the minute best practice. For example, keyword research will need to include consideration of Instant.

This should in theory reduce the number of impressions from repeated queries as users can now adjust their query readily without clicking through results, reducing or increasing the number of visits from short and long tail queries depending on how specific the results are from any given query.

How about PPC?

PPC Will certainly be affected by this update, as users can scan more results without clicking, and three second results, where ads are shown before a query is completed will increase impressions for Ads.

Keyword research should now include consideration for shortened versions of queries, including negative keywords for words which have different meanings in shortened versions.

So potentially higher impressions to be had but from a potentially reduced set of keywords as users are channelled to their goal by more relevant results. Remember also that this is from the same group of users, so the Cost Per Click (CPC) price also looks set to increase as a result of these changes.

Oh, and in case you are wondering – For now, SEO is not dead

Skip to start of post

Introduction

Note: If you are unable to view the Arabic and Cyrillic letters in this page you may need to install the required fonts.

Now that the first Internationalized Domain Names (IDNs) have gone live and have had some time to get established, it seems like a good time to revisit the finding of my previous article on IDNs “Can search engines handle Internationalized Domain Names (IDNs)?”

IDNs went live initially for three countries, all using the Arabic alphabet: Egypt (مصر); Saudi Arabia (السعودية); and the United Arab Emirates (امارات). Russia’s new IDN (рф) went live a little later, adding the Cyrillic alphabet to the mix, and additional IDNs have been created for other countries and alphabets. For this article I’ll take a look at how search engines handle these four IDNs.

IDN site search

To get an idea of how extensively the search engines have indexed sites on these new IDNs I’m going to use the “site:” operator. Although this operator is primarily used for finding pages on a particular website, e.g. [site:lbi.co.uk] it can also work all the way up to the TLD level, e.g. [site:uk].

Searching Google for [site:مصر], [site: السعودية], [site:امارات] and [site:рф] returns results from the IDNs  for Egypt, Saudi Arabia, the UAE and Russia as expected. Whilst the new IDN for Saudi Arabia only had 14 pages indexed when checked, the other IDNs all feature thousands of results.

Screenshot of a search for [site:рф] in Google:

Trying the same searches in Bing, however, does not return any results:

It appears that the site: operator does not work with these new IDNs in Bing (searching for other domains, e.g. [site:com], works as expected).

IDNs in search results?

The next area tested is whether the search engines will return these domains in their search results. To test this I picked out some random web pages on the new Egyptian IDN and tried searching for their title tags in both Google and Bing.

Searching both Google and Bing for the title of one web page, [مراكز التميز في البحث والتطوير - وزارة الإتصالات], brought up a number of web pages. The results from Google and Bing both contained a result from an IDN:

Google snippet featuring an IDN:

Bing snippet featuring an IDN:

More IDN bugs

Earlier I described how Bing’s site: operator does not yet work with IDNs. However, Google also has a number of IDN woes. Searching for [site:مصر] (the new IDN for Egypt) brings up the site سجل.مصر – however, clicking on the “Show more results from سجل.مصر” link in Google appears to be listing sites on domains other than سجل.مصر. Additionally, the “Show all results” link is percent encoded rather than listing the site name in the Arabic font.

In my previous look at how search engines handled IDNs I had found that Google’s links to “Translate this page” and “Cached” were broken for IDNs. Today it appears that Google has fixed the translation links – however, the cache links still do not appear to function.

Conclusion

The situation is much the same as it was back in February. The search engines can index websites which use IDNs – however, all of the major search engines still have bugs with their IDN support.

Given that the number of IDNs is set to grow and the number of websites using IDNs is likely to vastly increase in the near future, it’s vital that the search engines iron out the bugs in their IDN support. After all, if a search engine can’t handle websites from a particular properly, people might decide to switch to a search engine that can.

The “Binghoo” search alliance is finally coming to fruition. After some initial testing Yahoo! and Bing have announced that Yahoo! has completed the Bing transition and its search results are now being powered entirely by Bing.

This initial rollout only covers the US and the English-language version of Yahoo! in Canada, with other countries set to follow. Given the relative maturity of Bing in the UK compared to many other countries we would be surprised if the next rollout didn’t include the UK, although when this will happen is anyone’s guess. Yahoo! has said that the full worldwide rollout may be as late as 2012.

One country that might not be transitioning to Bing-Powered Yahoo! is Japan – the one country in the world where Yahoo! is a market leader. Yahoo! Japan is only partially owned by Yahoo! and has said that it is planning to use Google to power its search results instead of Bing, a move which Microsoft has slammed as anti-competitive.

Despite being late due to the Runaway tube train  and the Northern line being closed I managed to get to the Odeon in Covent Garden just on time (OK, I was 15 minutes late, but the session started 15 minutes after I arrived).

ScottGu in London Guathon DDD

Scott Guthrie opened up the event with an introduction to the sessions of the day. These sessions spanned four different aspects of Microsoft’s new technologies:

1. VS 2010 and ASP.NET 4
2. ASP.NET MVC 2
3. Windows Phone 7 Development
4. Web Development Futures: ASP.NET MVC 3, SQL CE and IIS Express

What was cool?

First session had many tips and tricks from Scott ‘Gu’ including some which are guaranteed to make the geeks squeal with delight.
The second session was the real meat in the sandwich for those who care about content and the way it is presented. For the past 10 years, ASP.Net was a great platform in terms of development productivity and for building enterprise applications. But its weakness was in the poor quality front-end code it generated; something that LBi interface developers and SEO consultants constantly complained about. However, Microsoft’s new ASP.NET MVC will be a game changer in this part (there are useful packages in IIS 7 to help on the SEO side.)

And yes, Windows Phone 7 will be released around October in UK. Microsoft intends to send a strong statement of intent to the smart phones market. Mike Ormond’s slides on the day can be found here.

And finally, In the last session on Web futures Scott Gu announced new technologies including IIS Express, MVC3, code-first Entity Framework, Razor (the new view engine for ASP.NET) that is going to be released soon by Microsoft. Then he discussed the common themes of these new developments which are to keep the existing technologies working better and cleaner and to enable extensibility and customization.

Revolutionary… No, but Microsoft are moving in the right direction whilst correcting some of the legacy issues of .Net. Who knows one day we may be able to have more than one form again…

Microsoft continue to pick up many of the good ideas that have been implemented in other languages for a little while and we’re starting to see some real productivity improvements as a consequence. However Microsoft remains a big beast and how these developments relate to the Sharepoints and Silverlights of the world remains to be seen. Refinement rather than reinvention is definitely order of the day for Microsoft and for that they should be commended.

As you can see below, at the time of writing, a search for [lbi.com] in google.co.uk will display a result for the Leo Baeck Institute in New York, a site about the history and culture of German speaking Jewry hosted on the domain ‘lbi.org’. The ‘sitelinks’ underneath the top result also erroneously refer and link to pages on the lbi.org domain:

This is badly wrong. As it happens, this is not a major disaster for LBi, but it could be much different for our natural search clients, who could lose significant revenues as a result of this kind of error.

So why did this happen?

There are no configurations or logical connections between the “lbi.com” site and the “lbi.org” site which could have mislead Google, leaving only two options; an error in Google code, or an error in a manually edited result – the latter of which we believe to be the most likely reason.

This is a very rare occurance that gives us an insight into the world of Google, in particular how some results are so well positioned, despite there being no ‘apparent’ reason for them to be performing so well.

We do see this from time to time, although it should be stressed that the overwhelming majority of sites will never see this kind of manual intervention, and usual best practices still apply.

One reason this result may have been singled out is due to Google’s recent focus on branded search. We suspect that brand results are one of the items currently being identified and prioritised by Google for search quality purposes.

Why would Google be manually editing search results in 2010?

Manually editing SERPS is more common than you might think. It happens for numerous reasons, from legal requests for removal of content, to handing out “black hat” SEO penalties, to delivering expected results for high volume navigational queries where, for example, a user is searching for a branded website.

Search engines have a conundrum, in that they need websites to be included in their index to attract searchers. If they remove websites for infringing terms and conditions no matter who they are, search engine users would soon get fed up and find another search engine. Likewise, if a search engine doesn’t surface expected results for a query because the site a user seeks is not optimised well enough to naturally be top of the search engine results, search engines reserve the right to manually edit results.

This introduces the potential for human error, which we believe is the case for the erroneous result demonstrated here.

Digging a little deeper:

The cached copy of this page, shown below as indexed on the 7th of August, clearly shows “lbi.com” in the cache URL, but “lbi.org” in the cache description. This is only the case for the homepage, for the phrase [lbi.com]:

The same error is evidenced with a search for [lbi.com] on the google.com site:

The same is also true for a “site:” operator search, which should only return pages from the “lbi.com” domain:

A search for [lbi] shows the expected results, including the correct ‘lbi.com’ homepage, so this is definitely included in the index:

The Leo Baeck Institute website (lbi.org) has no such error, showing that there is not a plain switch of site home pages:

We’ve dropped Google a line and will post further updates here when we hear any news back from them…

Update: Once we highlighted this, Google’s own John Mueller provided a response in the comments below, and within 24 hours the result for [lbi.com] has now been changed to display the expected results, with an LBi.com title, snippet and sitelinks appearing at the top of the page. We would like to extend our thanks to Google for ensuring a swift resolution.

Me is allegedly Google’s attempt to move on the ‘full service’ social network space that is Facebook (yes, I did just coin a social media description) but despite rumours proclaiming this as a major deal it is difficult not to be just a little bit cynical.  We have already seen Google launch both Wave and Buzz to ridiculous hype rapidly followed by almost laughable silence weeks after their respective launches – why should Me be any different? And more importantly, why is Google not focusing on joining up all of their various social hooks into something that makes sense? At present they have a variety of different social offerings yet most of them act like the others don’t exist – from Google Voice and Chat through to Buzz, Wave and even Google Reader (with its built in sharing settings) the graph may usually move between them but little else does.

So maybe that is all Google Me really is – a platform to pull together all of Google’s other platforms. Yet it is already being labelled a competitor to Facebook – this despite the fact that having to rebuild a whole new social graph on a new social network is about as enjoyable as actually being forced to converse with most of those forgotten school friends you could passively ignore before the days of Facebook.

So why bother? Well it probably has something to do with the fact that Facebook overtook Google in the US earlier this year to become the biggest site in terms of visits and it shows no sign of slowing. Whilst Google’s core offering (adverts served against search results) doesn’t currently directly compete with Facebook’s (adverts served against personal content) Google have to be more than just a little bit conscious that it wouldn’t take much for Facebook to make a move into their space.

What makes Google great? They have vast amounts of data about sites, the relationships between sites and the ways in which people access those sites. And what do Facebook have? Vast amounts of data about people, the relationships between people and, since the introduction of the ‘like’ button, the ways in which people access and share sites.

Facebook have recently started including sites with ‘like’ functionality into the search results a user receives when they search for anything on the Facebook site. But to be brutally honest, it’s horrible – there is no relevance to the results and it doesn’t fit with the user behaviour for people on the site. Yet it isn’t inconceivable that Facebook could buy a search engine and if you began to lay social graph data combined with content consumption habits you could have the next evolution of search: results that are socially aware. Imagine a result page where the sites your friends visit frequently get a little boost in the results for your searches.

The social search engine has seemed an obvious next step for years and yet still hasn’t happened – probably partly because no single company has had the relevant data sets, they have typically sat in separate businesses.  Of course let’s not forget that privacy concerns are likely to be a huge factor too, since search is just so personal. Yet packaged in the right way, whereby both sharing and privacy controls are simple and straightforward, it becomes a tantalising prospect.

Facebook have said search isn’t their focus (but they would, wouldn’t they) yet Google’s continued focus on building relationship data certainly suggests that social search may be the future.

Ultimately Google Me will still crash and burn if it can’t offering something unique that Facebook doesn’t – trying to move a population of 400 million to a new home is no mean feat – but if Google spies a threat to their core search business then you can be sure they are about to throw everything they can at the social space.

This story was originally posted at The Wall.

Microsoft’s search engine wasn’t always called “Bing” and its web crawler, “msnbot”, hasn’t kept up with the name change. When Microsoft renamed Live Search (formerly MSN Search) Bing, we have to admit to being mildly disappointed that it didn’t take the opportunity to rename its spider “Bing Bot”.

There are many good reasons not to change the name of a spider, especially one as widely used as Microsoft’s search spider. Many software packages look at the name of visiting browsers and spiders (known as the User-Agent) to perform a variety of functions, and it’s possible that problems might occur for a time on less well-configured websites if this were to be changed. For example, Yahoo! maintained the User-Agent “Slurp” for its spider, which it inherited from its acquisition of Inktomi, to “ensure consistency and minimal disruption”.

It appears that Microsoft has decided that the branding “Bing Bot” is too good to miss, however, and has announced that its next generation spider will indeed be renamed when it comes out of beta.

Here’s what site owners need to know:

When is this happening?

This will happen on 1^st October 2010.

This is also when Microsoft’s new spider will officially come out of beta.

What will the User-Agent be?

Microsoft’s current User-Agent is:

msnbot/2.0b (+http://search.msn.com/msnbot.htm)

The new Bing Bot User-Agent will be:

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

In addition to the “bingbot” branding, there are two other changes to note. Firstly, Microsoft is switching to the “Mozilla/5.0”-style User-Agent. Google made this change more than six years ago because it wanted web servers to treat its spider more like a real web browser. The second, more minor, change is that the “b” (meaning “beta”) in its version number has been dropped.

Any other changes to the spider’s requests?

In addition to the User-Agent change, Microsoft has also change the “From:” HTTP header field, so the old value of:

From: msnbot(at)microsoft.com

will become:

From: bingbot(at)microsoft.com

Will my old robots.txt entries still work?

Thankfully, Microsoft has decided to make its spider respect the User-Agent field which it currently recognises in robots.txt, “msnbot”. However, the way in which it will work from October is somewhat subtle, so deserves a brief explanation.

Whilst existing directives will still work, Microsoft is also going to recognise a “User-Agent:” robots.txt entry of “bingbot”, and it will give precedence to an entry of “bingbot” over an entry of “msnbot” (which, in turn, has precedence over the catch-all User-Agent entry of “*”). This means that, if you add robots.txt rules for “bingbot”, it will ignore all other rules, including those for “msnbot”.

Whilst adding conflicting “msnbot” and “bingbot” entries hopefully isn’t too likely to happen on most sites, in a larger, more complex organisation in which many different people or departments are able to make changes to robots.txt files, I wouldn’t be surprised to see someone accidentally trip up and add a new “bingbot” entry which doesn’t match up with the already existing “msnbot” entry (for example, where a separate “crawl-delay” value for Bing is specified).

Microsoft clearly wants site owners to update their robot.txt files with the new User-Agent, and we’d definitely recommend that you do this – but don’t forget that the new Bing Bot only launches on 1^st October – until then, you should still use the old “msnbot” terminology in your robots.txt files.

What should I do now?

Firstly, if you currently have a separate robots.txt entry for msnbot on your site(s), make a note on your calendar on to change it to “bingbot” on October 1^st.

Secondly, make sure that your website doesn’t do anything else special for Microsoft’s crawler or for visitors which don’t identify themselves as ‘Mozilla compatible’. This could include tools such as analytics packages or software which performs anti-spam functionality such as request rate-limiting.

Other than that, there shouldn’t be anything to worry about! However, in the (hopefully unlikely) event that you do experience any problems come October, Microsoft has set up an email address (bingbot@microsoft.com) to help to resolve any issues.

“Caffeine provides 50 percent fresher results for web searches than our last index, and it’s the largest collection of web content we’ve offered.”

Google’s head of spam also explained the update at an SMX advanced session captured on video for Search Engine Land. Matt’s key points in summary were:
Caffeine…

• Instead of crawling millions of documents in one day and then pushing it live hours later – with the caffeine update  Google can crawl documents and immediately put them into the index to be served live seconds later. So the entire index becomes closer to real time.

• Increases Google’s ability to scale up the capacity of its index (In the official Google blog post it says that Caffeine already uses nearly 100 million gigabytes of storage!)

• Makes it easier for Google to annotate documents with information.

As this is an update to Google’s infrastructure, it should not affect rankings.

Following from last week’s post about video SEO, this week’s post covers how to distribute your videos across the web and track their performance.

Whilst you can host video files on your own site and submit them to video search engines (Google, Yahoo!, Bing, Blinkx etc.), YouTube is the 800 lb gorilla in the video space. The only way in which your video will be found by people searching on the YouTube website is if you upload the video to YouTube itself.

Using YouTube also has additional benefits – YouTube automatically creates Media RSS feeds, which you can use to submit the video to search engines, and it also makes hosting of videos effectively “free”. However, the Media RSS feeds that YouTube provides link back to the YouTube page, not to the page on your site.

Therefore, we recommend hosting your videos on your own site as well as on YouTube. You can then generate your own Media RSS feeds (linking back to your site) and submit these feeds, rather than the YouTube feeds, to the various video search engines. If hosting videos on your own site, it may also be useful to provide the video content in multiple formats – the more formats in which the video is available, the larger the potential audience (although more formats also means additional costs in both time and bandwidth, so there is a definite trade-off involved).

It may be worth uploading the video to other video hosting platforms, such as Dailymotion, MetaCafe or Vimeo, as well as to YouTube.

When uploading your video to video hosting sites like these (as opposed to submitting it to video search engines), we recommend watermarking the video with the brand name to prevent it from being re-used without attribution.

Most video hosting sites allow you to include a URL along with each video – each video that you upload to a third-party site should ideally link back to the page on your site on which the video is hosted.

It is also a good idea to embed the URL of the page on your site where the video file is included within the video itself. Short URLs are generally better, as users will have to manually type them in. URL shortening services which support tracking of users are particular useful here: they can allow you to track users who visit your site after watching one of your videos, and identify which of your videos and which video hosting sites are attracting the most visitors.