Re: Byte ranges (actually robots) from Gavin Nicol on 1995-05-25 (www-talk@w3.org from May to June 1995)

From: Gavin Nicol <gtn@ebt.com>
Date: Thu, 25 May 1995 11:57:15 -0400
To: nazgul@utopia.com
Cc: www-talk@www10.w3.org
Message-Id: <199505251557.LAA21338@ebt-inc.ebt.com>

>>Isn't there a standard permissions file that robots can fetch?
> 
>Yes, but that just let's me tell them what to stay away from. The issue
>here is not that they should stay away, but that I should recognize them.

Fair enough. I'd also like to recognise them, and some kind of
convention in User-Agent or somesuch seems desirable. 

Putting a robot on our server is a great way of stress-testing it ;-)

Received on Thursday, 25 May 1995 11:54:54 UTC