- From: Mudry Julien <julien.mudry@elca.ch>
- Date: Thu, 24 Oct 2002 05:11:33 -0400 (EDT)
- To: "'Shantz'" <michaelshantz@attbi.com>
- Cc: "Jigsaw List (E-mail)" <www-jigsaw@w3.org>
Hello The robots.txt file allows a webmaster to exclude some pages or directories from browsing by webcrawlers. It's a standard called "Standard for Robot Exclusion". You can get more information regarding it here: http://www.robotstxt.org/ Specifically, to answer your question: http://www.robotstxt.org/wc/faq.html#log Regards, Julien > -----Original Message----- > From: Shantz [mailto:michaelshantz@attbi.com] > Sent: Thursday, October 24, 2002 10:57 AM > To: www-jigsaw@w3.org > Subject: robots.txt? off-topic > > > > > > > I've been using jigsaw to serve a webpage for a while. > When looking at the log, I often see what appears to be webcrawlers > doing a GET on robots.txt. I have never had such a file. Does anyone > know what this is about? > > Mike > >
Received on Thursday, 24 October 2002 08:45:31 UTC