Re: Just what *does* robots.txt mean for a LOD site?

• From: Tobias Käfer <tobias.kaefer@kit.edu>
• Date: Tue, 5 Aug 2014 18:33:03 +0200
• To: <public-lod@w3.org>
• Message-ID: <53E1073F.2010507@kit.edu>

Hi Hugh,

 > By the way, have I got my robots.txt right?
> In particular, is the
> User-agent: LDSpider
> correct?
> Should I worry about case-sensitivity?

The library (norbert) that is employed in LDspider is case-insensitive 
for the user agent. The user agent that is sent is "ldspider".

I suppose you want ldspider to crawl your site (highly appreciated), so 
you should change the line in your robots.txt for LDspider to:
a) Disallow:
b) Allow: /
And not leave it with:
c) Allow: *
The star there does not bring the desired behaviour (and I have not 
found it in the spec for the path either), in fact, it keeps LDspider 
from crawling the folders you specified for exclusion for the other 
crawlers.

Cheers,

Tobias

Received on Tuesday, 5 August 2014 16:33:34 UTC