RE: robots.txt? off-topic

Hello

The robots.txt file allows a webmaster to exclude some
pages or directories from browsing by webcrawlers. It's
a standard called "Standard for Robot Exclusion". You
can get more information regarding it here: 
	http://www.robotstxt.org/

Specifically, to answer your question:
	http://www.robotstxt.org/wc/faq.html#log

Regards,

	Julien

> -----Original Message-----
> From: Shantz [mailto:michaelshantz@attbi.com]
> Sent: Thursday, October 24, 2002 10:57 AM
> To: www-jigsaw@w3.org
> Subject: robots.txt? off-topic
> 
> 
> 
> 
> 
> 
> I've been using jigsaw to serve a webpage for a while.
> When looking at the log, I often see what appears to be webcrawlers
> doing a GET on robots.txt.  I have never had such a file.  Does anyone
> know what this is about?
> 
> Mike
> 
> 

Received on Thursday, 24 October 2002 08:45:31 UTC