W3C home > Mailing lists > Public > www-validator@w3.org > July 2007

Re: [checklink] How to override robots.txt?

From: Ville Skyttä <ville.skytta@iki.fi>
Date: Wed, 18 Jul 2007 19:45:12 +0300
To: www-validator@w3.org
Message-Id: <200707181945.12750.ville.skytta@iki.fi>

On Tuesday 17 July 2007, Stephan Windmüller wrote:

> Is it possible to override or disable the robots.txt-feature?

Not from the link checker side without modifying the code.  Perhaps the target 
site administrators would be willing to allow the link checker to crawl it?  
More info: http://validator.w3.org/docs/checklink.html#bot

> If not, how can I prevent checklink to list these adresses as errors when
> using "-q"?

I'm afraid either modifying the link checker code or postprocessing its output 
eg. with sed or perl are the only ways it can be currently done.
Received on Wednesday, 18 July 2007 16:46:15 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:25 GMT