Apr 14

Written by: Steve Gray
4/14/2009 10:20 AM 

The search engines use the robots.text file in the root of a web site to (sort of) determine what content they should catalog. I'm not an expert in how well that's respected, but I always include one because I get 404 errors on the site if I don't

http://www.robotstxt.org/robotstxt.html is a good reference for the subject.

Personally, I just use this:

User-agent: *
Disallow: