- From: Etienne Miret <elimerl@gmail.com>
- Date: Tue, 23 Sep 2008 20:33:16 +0200
- To: Ridvan <ridvan@fikp.org>
- CC: W3C Validator Community <www-validator@w3.org>
Hello,
The linkchecker sends the following User-agent field:
> User-Agent: W3C-checklink/4.3 [4.42] libwww-perl/5.805
and it seems that your server is configured to deny access to any
user-agent that identify itself as "libwww-perl". Hence, linkchecker is
denied access to your website.
As you asked for it, the IP from which comes linkchecker request is:
128.30.52.13. At least, that’s the IP it used last time it checked my
site (search your logs for W3C-Checklink).
Additionally, I suggest you review your robots.txt file:
- It is sent as 'text/html', but the normal media-type for this file is
'text/plain'.
- The last three records don’t follow any "User-Agent" record, hence if
I understand the standard correctly, they don’t apply to any robots.
Regards,
Etienne
Received on Tuesday, 23 September 2008 18:34:02 UTC