W3C home > Mailing lists > Public > www-jigsaw@w3.org > September to October 2002

robots.txt? off-topic

From: Shantz <michaelshantz@attbi.com>
Date: Wed, 23 Oct 2002 12:35:33 -0400 (EDT)
Message-ID: <018b01c27ab2$c10828e0$0501a8c0@Lindow>
To: <www-jigsaw@w3.org>




I've been using jigsaw to serve a webpage for a while.
When looking at the log, I often see what appears to be webcrawlers
doing a GET on robots.txt.  I have never had such a file.  Does anyone
know what this is about?

Mike
Received on Thursday, 24 October 2002 04:57:21 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 9 April 2012 12:13:36 GMT