- From: Hugh Glaser <hugh@glasers.org>
- Date: Wed, 30 Jul 2014 18:59:33 +0100
- To: Linking Open Data <public-lod@w3.org>
Thanks all. OK, I can live with that. So things like Tabulator, Sig.ma and SemWeb Browsers can be expected to go through a general robots.txt Disallow, which is what I was hoping. Yes, thanks Aidan, I know I can do various User-agents, but I really just wanted to stop anything like googlebot. By the way, have I got my robots.txt right? http://ibm.rkbexplorer.com/robots.txt In particular, is the User-agent: LDSpider correct? Should I worry about case-sensitivity? Thanks again, all. Hugh On 27 Jul 2014, at 19:23, Gannon Dick <gannon_dick@yahoo.com> wrote: > > -------------------------------------------- > On Sat, 7/26/14, ahogan@dcc.uchile.cl <ahogan@dcc.uchile.cl> wrote: > > The difference in opinion remains to what extent Linked Data > agents need to pay attention to the robots.txt file. > > As many others have suggested, I buy into the idea of any > agent not relying document-wise on user input being subject to > robots.txt. > > ===================== > +1 > Just a comment. > > Somewhere, sometime, somebody with Yahoo Mail decided that public-lod mail was spam, so every morning I dig it out because I value the content. > > Of course, I could wish for a Linked Data Agent which does that for me, but that would be to complete a banal or vicious cycle, depending on the circle classification scheme in use. I'm looking gor virtuous cycles and in the case of robots.txt, "The lady doth protest too much, methinks". > --Gannon > > > -- Hugh Glaser 20 Portchester Rise Eastleigh SO50 4QS Mobile: +44 75 9533 4155, Home: +44 23 8061 5652
Received on Wednesday, 30 July 2014 18:00:14 UTC