- From: Steve Harris <steve.harris@garlik.com>
- Date: Wed, 22 Jun 2011 08:00:40 +0100
- To: Martin Hepp <martin.hepp@ebusiness-unibw.org>
- Cc: Daniel Herzig <herzig@kit.edu>, <semantic-web@w3.org>, <public-lod@w3.org>
While I don't agree with Andreas exactly that it's the site owners fault, this is something that publishers of non-semantic data have to deal with. If you publish a large collection of interlinked data which looks interesting to conventional crawlers and is expensive to generate, conventional web crawlers will be all over it. The main difference is that a greater percentage of those are written properly, to follow robots.txt and the guidelines about hit frequency (maximum 1 request per second per domain, no parallel crawling). Has someone published similar guidelines for semantic web crawlers? The ones that don't behave themselves get banned, either in robots.txt, or explicitly by the server. - Steve On 2011-06-22, at 06:07, Martin Hepp wrote: > Hi Daniel, > Thanks for the link! I will relay this to relevant site-owners. > > However, I still challenge Andreas' statement that the site-owners are to blame for publishing large amounts of data on small servers. > > One can publish 10,000 PDF documents on a tiny server without being hit by DoS-style crazy crawlers. Why should the same not hold if I publish RDF? > > But for sure, it is necessary to advise all publishers of large RDF datasets to protect themselves against hungry crawlers and actual DoS attacks. > > Imagine if a large site was brought down by a botnet that is exploiting Semantic Sitemap information for DoS attacks, focussing on the large dump files. > This could end LOD experiments for that site. > > > Best > > Martin > > > On Jun 21, 2011, at 10:24 AM, Daniel Herzig wrote: > >> >> Hi Martin, >> >> Have you tried to put a Squid [1] as reverse proxy in front of your servers and use delay pools [2] to catch hungry crawlers? >> >> Cheers, >> Daniel >> >> [1] http://www.squid-cache.org/ >> [2] http://wiki.squid-cache.org/Features/DelayPools >> >> On 21.06.2011, at 09:49, Martin Hepp wrote: >> >>> Hi all: >>> >>> For the third time in a few weeks, we had massive complaints from site-owners that Semantic Web crawlers from Universities visited their sites in a way close to a denial-of-service attack, i.e., crawling data with maximum bandwidth in a parallelized approach. >>> >>> It's clear that a single, stupidly written crawler script, run from a powerful University network, can quickly create terrible traffic load. >>> >>> Many of the scripts we saw >>> >>> - ignored robots.txt, >>> - ignored clear crawling speed limitations in robots.txt, >>> - did not identify themselves properly in the HTTP request header or lacked contact information therein, >>> - used no mechanisms at all for limiting the default crawling speed and re-crawling delays. >>> >>> This irresponsible behavior can be the final reason for site-owners to say farewell to academic/W3C-sponsored semantic technology. >>> >>> So please, please - advise all of your colleagues and students to NOT write simple crawler scripts for the billion triples challenge or whatsoever without familiarizing themselves with the state of the art in "friendly crawling". >>> >>> Best wishes >>> >>> Martin Hepp >>> >> > > -- Steve Harris, CTO, Garlik Limited 1-3 Halford Road, Richmond, TW10 6AW, UK +44 20 8439 8203 http://www.garlik.com/ Registered in England and Wales 535 7233 VAT # 849 0517 11 Registered office: Thames House, Portsmouth Road, Esher, Surrey, KT10 9AD
Received on Wednesday, 22 June 2011 07:01:10 UTC