- From: David Dorward <david@dorward.me.uk>
- Date: Mon, 23 May 2005 17:23:17 +0100
- To: www-validator@w3.org
On Mon, May 23, 2005 at 11:18:30AM -0500, Tim Burkhart wrote: > Is there a way to disable robots in version 4.1? Why would you want to? It has a readily identifiable User Agent string, so you can explicitly allow it in your own robots.txt. On sites you don't control it isn't very nice to ignore the "Keep Out" request. -- David Dorward http://dorward.me.uk
Received on Monday, 23 May 2005 16:23:20 UTC