W3C home > Mailing lists > Public > www-validator@w3.org > July 2007

Re: [checklink] How to override robots.txt?

From: Stephan Windmüller <windy@white-hawk.de>
Date: Wed, 18 Jul 2007 11:28:03 +0200
To: www-validator@w3.org
Message-ID: <20070718092803.GB14735@speutel.de>
On Wed, 18. Jul 2007, Michael Adams wrote:

> > We want to use checklink to search for broken links on our site. Of
> > course there should be only a warning mail when something went
> > wrong.  Unfortunately this is also true when external links are
> > disallowed by a robots.txt.
> Why not FTP into your site and download then delete robots.txt
> temporarily? Upload again to restore when finished.

The problem is not our robots.txt but those of external sites.

- Stephan

Received on Wednesday, 18 July 2007 09:28:32 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 7 January 2015 15:30:55 UTC