W3C home > Mailing lists > Public > www-validator@w3.org > May 2005

Re: checklink: Disable Robots

From: David Dorward <david@dorward.me.uk>
Date: Mon, 23 May 2005 17:23:17 +0100
To: www-validator@w3.org
Message-ID: <20050523162317.GB26805@us-lot.org>

On Mon, May 23, 2005 at 11:18:30AM -0500, Tim Burkhart wrote:
> Is there a way to disable robots in version 4.1?

Why would you want to? It has a readily identifiable User Agent
string, so you can explicitly allow it in your own robots.txt. On
sites you don't control it isn't very nice to ignore the "Keep Out"

David Dorward                                      http://dorward.me.uk
Received on Monday, 23 May 2005 16:23:20 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 22:58:51 UTC