W3C home > Mailing lists > Public > www-validator@w3.org > May 2005

Re: checklink: Disable Robots

From: David Dorward <david@dorward.me.uk>
Date: Mon, 23 May 2005 17:23:17 +0100
To: www-validator@w3.org
Message-ID: <20050523162317.GB26805@us-lot.org>

On Mon, May 23, 2005 at 11:18:30AM -0500, Tim Burkhart wrote:
> Is there a way to disable robots in version 4.1?

Why would you want to? It has a readily identifiable User Agent
string, so you can explicitly allow it in your own robots.txt. On
sites you don't control it isn't very nice to ignore the "Keep Out"
request.

-- 
David Dorward                                      http://dorward.me.uk
Received on Monday, 23 May 2005 16:23:20 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:19 GMT