W3C home > Mailing lists > Public > www-jigsaw@w3.org > September to October 2002

Re: robots.txt? off-topic

From: Matthew Baker <mcb499@ecs.soton.ac.uk>
Date: Thu, 24 Oct 2002 12:18:15 +0100
Message-ID: <001c01c27b4f$0c48b6c0$31474e98@ecs.soton.ac.uk>
To: "Shantz" <michaelshantz@attbi.com>
Cc: <www-jigsaw@w3.org>

It is a file that allows you to specify which parts of the site are not
visited by web crawlers.

See http://www.robotstxt.org/ for details.

Matt.

----- Original Message -----
From: "Shantz" <michaelshantz@attbi.com>
To: <www-jigsaw@w3.org>
Sent: Wednesday, October 23, 2002 5:35 PM
Subject: robots.txt? off-topic


>
>
>
>
> I've been using jigsaw to serve a webpage for a while.
> When looking at the log, I often see what appears to be webcrawlers
> doing a GET on robots.txt.  I have never had such a file.  Does anyone
> know what this is about?
>
> Mike
>
>
>
Received on Thursday, 24 October 2002 07:18:29 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 9 April 2012 12:13:36 GMT