W3C home > Mailing lists > Public > www-validator@w3.org > May 2005

Re: checklink: Disable Robots

From: Tim Burkhart <tmb5858@ksu.edu>
Date: Mon, 23 May 2005 11:54:30 -0500
Message-ID: <42920AC6.2030400@ksu.edu>
To: www-validator@w3.org

Hi,

I was wanting to disable robots because we have a huge website and are 
wanting to run a link checker throughout the entire site to verify that 
all of the links work. However, we want to ensure that no other robots 
are allowed to access some of our files, so we still keep all of our 
files and folders in robots.txt.

-Tim Burkhart
Received on Monday, 23 May 2005 16:54:35 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:19 GMT