Re: checklink: Disable Robots

On Mon, May 23, 2005 at 11:18:30AM -0500, Tim Burkhart wrote:
> Is there a way to disable robots in version 4.1?

Why would you want to? It has a readily identifiable User Agent
string, so you can explicitly allow it in your own robots.txt. On
sites you don't control it isn't very nice to ignore the "Keep Out"

David Dorward                            

Received on Monday, 23 May 2005 16:23:20 UTC