W3C home > Mailing lists > Public > www-validator@w3.org > May 2005

Re: checklink: Disable Robots

From: Tim Burkhart <tmb5858@ksu.edu>
Date: Mon, 23 May 2005 11:54:30 -0500
Message-ID: <42920AC6.2030400@ksu.edu>
To: www-validator@w3.org


I was wanting to disable robots because we have a huge website and are 
wanting to run a link checker throughout the entire site to verify that 
all of the links work. However, we want to ensure that no other robots 
are allowed to access some of our files, so we still keep all of our 
files and folders in robots.txt.

-Tim Burkhart
Received on Monday, 23 May 2005 16:54:35 UTC

This archive was generated by hypermail 2.3.1 : Friday, 27 November 2015 11:01:25 UTC