Re: checklink: Disable Robots from Tim Burkhart on 2005-05-23 (www-validator@w3.org from May 2005)

From: Tim Burkhart <tmb5858@ksu.edu>
Date: Mon, 23 May 2005 11:54:30 -0500
To: www-validator@w3.org
Message-ID: <42920AC6.2030400@ksu.edu>

Hi,

I was wanting to disable robots because we have a huge website and are 
wanting to run a link checker throughout the entire site to verify that 
all of the links work. However, we want to ensure that no other robots 
are allowed to access some of our files, so we still keep all of our 
files and folders in robots.txt.

-Tim Burkhart

Received on Monday, 23 May 2005 16:54:35 UTC