W3C home > Mailing lists > Public > www-jigsaw@w3.org > September to October 2002

RE: robots.txt? off-topic

From: Mudry Julien <julien.mudry@elca.ch>
Date: Thu, 24 Oct 2002 05:11:33 -0400 (EDT)
Message-ID: <A13B47F8EC3FD111A91500C04FD2A282033B7658@exchange.elca.ch>
To: "'Shantz'" <michaelshantz@attbi.com>
Cc: "Jigsaw List (E-mail)" <www-jigsaw@w3.org>




Hello

The robots.txt file allows a webmaster to exclude some
pages or directories from browsing by webcrawlers. It's
a standard called "Standard for Robot Exclusion". You
can get more information regarding it here: 
	http://www.robotstxt.org/

Specifically, to answer your question:
	http://www.robotstxt.org/wc/faq.html#log

Regards,

	Julien

> -----Original Message-----
> From: Shantz [mailto:michaelshantz@attbi.com]
> Sent: Thursday, October 24, 2002 10:57 AM
> To: www-jigsaw@w3.org
> Subject: robots.txt? off-topic
> 
> 
> 
> 
> 
> 
> I've been using jigsaw to serve a webpage for a while.
> When looking at the log, I often see what appears to be webcrawlers
> doing a GET on robots.txt.  I have never had such a file.  Does anyone
> know what this is about?
> 
> Mike
> 
> 
Received on Thursday, 24 October 2002 08:45:31 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 9 April 2012 12:13:36 GMT