Technology with attitude

New White House Website Open To Search Engines

1

Web guru Jason Kottke points out that the Bush administration had almost 2,400 different rules governing what search engines could crawl on the site and report in their results. This was most likely done so that no caches could be created to later compare with any changes that were made.

But seriously, most of them aren’t necessary and some of them are just plain odd.

For instance, the White House didn’t want search engines crawling pages about Bush’s dog Barney?

Disallow: /barney/barney/text
Disallow: /barney/barneycam/text
Disallow: /barney/newmedia/text
Disallow: /barney/photoessay/beazley/text
Disallow: /barney/photoessay/text
Disallow: /barney/photoessay2/text
Disallow: /barney/text
Disallow: /barneycam/text

Or how about the pages about Easter?

Disallow: /easter/2002/eggsbystate/text
Disallow: /easter/2002/text
Disallow: /easter/2003/eggsbystate/text
Disallow: /easter/2003/text
Disallow: /easter/2004/eggsbystate/text
Disallow: /easter/2004/media/text
Disallow: /easter/2004/text
Disallow: /easter/2005/eggsbystate/text
Disallow: /easter/2005/photoessay/text
Disallow: /easter/2005/text
Disallow: /easter/2005/video/text
Disallow: /easter/2006/eggsbystate/text
Disallow: /easter/2006/photoessay/text
Disallow: /easter/2006/text
Disallow: /easter/2006/video/text
Disallow: /easter/2007/eggsbystate/text
Disallow: /easter/2007/photoessay/easter-2007/text
Disallow: /easter/2007/photoessay/photoessays/text
Disallow: /easter/2007/photoessay/text
Disallow: /easter/2007/text
Disallow: /easter/2007/video/text
Disallow: /easter/2008/eastervideo/text
Disallow: /easter/2008/eggsbystate/text
Disallow: /easter/2008/photoessay/preparing/text
Disallow: /easter/2008/photoessay/text
Disallow: /easter/2008/text
Disallow: /easter/2008/video/text
Disallow: /easter/states/text
Disallow: /easter/text

On the other hand, Obama’s new site has one rule for search engines…

Disallow: /includes/

Perhaps this will change, but it’s an encouraging sign nonetheless.