Re: Just what *does* robots.txt mean for a LOD site?

Thanks all.
OK, I can live with that.

So things like Tabulator, Sig.ma and SemWeb Browsers can be expected to go through a general robots.txt Disallow, which is what I was hoping.

Yes, thanks Aidan, I know I can do various User-agents, but I really just wanted to stop anything like googlebot.

By the way, have I got my robots.txt right?
http://ibm.rkbexplorer.com/robots.txt
In particular, is the
User-agent: LDSpider
correct?
Should I worry about case-sensitivity?

Thanks again, all.
Hugh


On 27 Jul 2014, at 19:23, Gannon Dick <gannon_dick@yahoo.com> wrote:

> 
> --------------------------------------------
> On Sat, 7/26/14, ahogan@dcc.uchile.cl <ahogan@dcc.uchile.cl> wrote:
> 
> The difference in opinion remains to what extent Linked Data
> agents need to pay attention to the robots.txt file.
> 
> As many others have suggested, I buy into the idea of any
> agent not relying document-wise on user input being subject to
> robots.txt.
> 
> =====================
> +1
> Just a comment.
> 
> Somewhere, sometime, somebody with Yahoo Mail decided that public-lod mail was spam, so every morning I dig it out because I value the content.
> 
> Of course, I could wish for a Linked Data Agent which does that for me, but that would be to complete a banal or vicious cycle, depending on the circle classification scheme in use.  I'm looking gor virtuous cycles and in the case of robots.txt, "The lady doth protest too much, methinks".
> --Gannon
> 
> 
> 

-- 
Hugh Glaser
   20 Portchester Rise
   Eastleigh
   SO50 4QS
Mobile: +44 75 9533 4155, Home: +44 23 8061 5652

Received on Wednesday, 30 July 2014 18:00:14 UTC