W3C home > Mailing lists > Public > www-talk@w3.org > May to June 1995

Re: Byte ranges (actually robots)

From: Gavin Nicol <gtn@ebt.com>
Date: Thu, 25 May 1995 11:57:15 -0400
Message-Id: <199505251557.LAA21338@ebt-inc.ebt.com>
To: nazgul@utopia.com
Cc: www-talk@www10.w3.org
>>Isn't there a standard permissions file that robots can fetch?
> 
>Yes, but that just let's me tell them what to stay away from. The issue
>here is not that they should stay away, but that I should recognize them.

Fair enough. I'd also like to recognise them, and some kind of
convention in User-Agent or somesuch seems desirable. 

Putting a robot on our server is a great way of stress-testing it ;-)
Received on Thursday, 25 May 1995 11:54:54 UTC

This archive was generated by hypermail 2.4.0 : Monday, 20 January 2020 16:08:17 UTC