Re: Byte ranges (actually robots) from Gavin Nicol on 1995-05-25 (www-talk@w3.org from May to June 1995)

From: Gavin Nicol <gtn@ebt.com>
Date: Thu, 25 May 1995 09:21:10 -0400
To: nazgul@utopia.com
Cc: www-talk@www10.w3.org
Message-Id: <199505251321.JAA13703@ebt-inc.ebt.com>

>So what to do? If we recognize the user-agent as a known robot, we give it
>a flat presentation of the structure.  The catch of course, is that there
>is no standard way to recognize a robot.  Some convention in the UserAgent
>field would be sufficient, but....

Isn't there a standard permissions file that robots can fetch?

Received on Thursday, 25 May 1995 09:18:54 UTC