W3C home > Mailing lists > Public > public-lod@w3.org > June 2010

Re: Please stop massive crawling against http://openean.kaufkauf.net/id/

From: Robert Fuller <robert.fuller@deri.org>
Date: Tue, 08 Jun 2010 13:14:19 +0100
Message-ID: <4C0E341B.6020204@deri.org>
To: public-lod@w3.org
Hi,

Sindice clearly identifies itself in the user agent http header. 
Currently we use these user agents:

1. "Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 
+http://sindice.com/developers/bot)"

2. "SindiceFetcher/Ping Manager (http://sindice.com/developers/bot"

3. "sindice.net ontology fetcher"

Niceness is implemented in our main fetcher. In some cases there may be 
bursts on sites providing distributed ontologies. Speaking with the 
group here it seems unlikely that we have not been hitting kaufkauf.net, 
  however if you can provide an IP address I can do some further 
verification.

I understand that http://lod.openlinksw.com/sparql is now hosted at 
DERI, and I wonder could some of the traffic be related to that? Again, 
if you can provide an IP address I will do some further verification.


Kind regards,
Rob.

--
Robert Fuller
Research Associate
DERI, Galway
Received on Tuesday, 8 June 2010 12:45:43 UTC

This archive was generated by hypermail 2.3.1 : Sunday, 31 March 2013 14:24:27 UTC