del.icio.us Bans Search Engine Spiders

by Colin Cochrane 2/16/2008 10:06:00 PM
It appears that within the past 2-3 days the popular social book-marking site del.icio.us has started blocking the major search engine spiders from crawling their site. This isn't a simple robots.txt exclusion, but rather a 404 response that is now being served based on the requesting User-Agent.

This raises some questions as to the intentions of del.icio.us, and perhaps Yahoo! With Yahoo! recently integrating del.icio.us bookmarks into its search results this could an attempt to enhance the effectiveness of that new feature by preventing competing search engines from indexing content from del.icio.us. [More]

Tags: , , ,

Search | Yahoo

Powered by BlogEngine.NET 2.5.0.6

All Content and Intellectual Property is under Copyright Protection | Colin Cochrane ©2007

About the author

Colin Cochrane

Colin Cochrane

SEO and ASP.NET Developer.

Recent comments

Recent posts

Archive

Authors

Disclaimer

This is a personal weblog. The opinions expressed here represent my own and not those of my employer. © Copyright Colin Cochrane 2014

Sign in