Re: Robots.txt with Link checker

On Tuesday 08 June 2010, David Dorward wrote:
> On 6 Jun 2010, at 22:50, webmaster@groenhuis.nu wrote:
> > As webmaster for the website http://www.groenhuis.nu i want our website
> > to validate. The link checker tool is "complaining" it can't acces some
> > javascript files. Is this something i should correct. I put these files
> > into the robots.txt because i think it is not nessesary for
> > searchengines to index these files.
> 
> Well, yes. You told robots not to go to your JS. The link checker is a
> robot and it is doing what you told it to do.

See also http://validator.w3.org/docs/checklink.html#bot

Received on Wednesday, 9 June 2010 18:59:52 UTC