Re: robots.txt? off-topic

Thanks to all for explaining this to me.
mike

----- Original Message ----- 
From: "Mudry Julien" <julien.mudry@elca.ch>
To: "'Shantz'" <michaelshantz@attbi.com>
Cc: "Jigsaw List (E-mail)" <www-jigsaw@w3.org>
Sent: Thursday, October 24, 2002 2:10 AM
Subject: RE: robots.txt? off-topic


> Hello
> 
> The robots.txt file allows a webmaster to exclude some
> pages or directories from browsing by webcrawlers. It's
> a standard called "Standard for Robot Exclusion". You
> can get more information regarding it here: 
> http://www.robotstxt.org/
> 
> Specifically, to answer your question:
> http://www.robotstxt.org/wc/faq.html#log
> 
> Regards,
> 
> Julien
> 
> > -----Original Message-----
> > From: Shantz [mailto:michaelshantz@attbi.com]
> > Sent: Thursday, October 24, 2002 10:57 AM
> > To: www-jigsaw@w3.org
> > Subject: robots.txt? off-topic
> > 
> > 
> > 
> > 
> > 
> > 
> > I've been using jigsaw to serve a webpage for a while.
> > When looking at the log, I often see what appears to be webcrawlers
> > doing a GET on robots.txt.  I have never had such a file.  Does anyone
> > know what this is about?
> > 
> > Mike
> > 
> >

Received on Thursday, 24 October 2002 13:53:21 UTC