Re: robots.txt? off-topic from Matthew Baker on 2002-10-24 (www-jigsaw@w3.org from September to October 2002)

From: Matthew Baker <mcb499@ecs.soton.ac.uk>
Date: Thu, 24 Oct 2002 12:18:15 +0100
To: "Shantz" <michaelshantz@attbi.com>
Cc: <www-jigsaw@w3.org>
Message-ID: <001c01c27b4f$0c48b6c0$31474e98@ecs.soton.ac.uk>

It is a file that allows you to specify which parts of the site are not
visited by web crawlers.

See http://www.robotstxt.org/ for details.

Matt.

----- Original Message -----
From: "Shantz" <michaelshantz@attbi.com>
To: <www-jigsaw@w3.org>
Sent: Wednesday, October 23, 2002 5:35 PM
Subject: robots.txt? off-topic


>
>
>
>
> I've been using jigsaw to serve a webpage for a while.
> When looking at the log, I often see what appears to be webcrawlers
> doing a GET on robots.txt.  I have never had such a file.  Does anyone
> know what this is about?
>
> Mike
>
>
>

Received on Thursday, 24 October 2002 07:18:29 UTC