W3C home > Mailing lists > Public > www-validator-cvs@w3.org > October 2004

[Bug 893] cache (non) existence

From: <bugzilla@wiggum.w3.org>
Date: Wed, 06 Oct 2004 02:00:00 +0000
To: www-validator-cvs@w3.org
Message-Id: <E1CF160-0006MY-SK@wiggum.w3.org>


ot@w3.org changed:

           What    |Removed                     |Added
         AssignedTo|ot@w3.org                   |ville.skytta@iki.fi
          Component|Other                       |checklink
            Product|CSSValidator                |LinkChecker
            Summary|excessive robot.txt probes  |cache (non) existence
            Version|CSS Validator               |4.0

------- Additional Comments From ot@w3.org  2004-10-06 02:00 -------
Most likely this is the Link Checker doing this, not the CSS validator.
Our site is not "probing" yours. 

Someone (possibly someone local to you, or someone with a site linking to yours) is certainly checking 
links to your site, and the link checker is following the robots exclusion protocol and doing your server 
a favor in doing so.

That said, a possible enhancement would be that the link checker cache the existence/lack of robots.txt 
for a given site instead of querying for it again and again.

Reassigning to proper product and owner.

------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
Received on Wednesday, 6 October 2004 02:00:01 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 23:02:07 UTC