W3C home > Mailing lists > Public > www-validator@w3.org > May 2005

Re: checklink: Disable Robots

From: Tim Burkhart <tmb5858@ksu.edu>
Date: Mon, 23 May 2005 12:08:03 -0500
Message-ID: <42920DF3.7060507@ksu.edu>
To: www-validator@w3.org


Sorry, I have never worked with the robots.txt file before. But I just 
looked at the documentation closer, and I realize that I misinterpreted 
how robots.txt works.

Whenever I took a first quick glance, I just noticed the lines:

Disallow: ...
Allow: ...

And I assumed they were for all robots. Now that I took a closer look, I 
see how to do this. Thanks for the help!

Received on Monday, 23 May 2005 17:08:08 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 7 January 2015 15:30:48 UTC