Re: checklink: User-Agent causes HTTP error 406

On Sunday 30 August 2009, Kevin Keane wrote:
> On a number of Web sites, check-link returns with an HTTP error 406,
> even with correct Accept header settings, and even when wget and links
> can access the site without a problem.
>
> Through trial and error, I found that on at least one site, the culprit
> is the User-Agent header. Adding the line
>
>   $request->header('User-Agent', 'W3C-linkcheck');
>
> before line 1307 solves this issue. Normally, the user agent is
> W3C-checklink/4.5 [4.154] libwww-perl/5.805

They might be triggering on "libwww-perl" in the user agent string.  I suggest 
contacting such sites and reporting the bug/asking for an explanation why.