W3C home > Mailing lists > Public > www-jigsaw@w3.org > September to October 2002

robots.txt? off-topic

From: Shantz <michaelshantz@attbi.com>
Date: Wed, 23 Oct 2002 12:35:33 -0400 (EDT)
Message-ID: <018b01c27ab2$c10828e0$0501a8c0@Lindow>
To: <www-jigsaw@w3.org>

I've been using jigsaw to serve a webpage for a while.
When looking at the log, I often see what appears to be webcrawlers
doing a GET on robots.txt.  I have never had such a file.  Does anyone
know what this is about?

Received on Thursday, 24 October 2002 04:57:21 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:25:38 UTC