Re: Please stop massive crawling against http://openean.kaufkauf.net/id/

Hi,

Sindice clearly identifies itself in the user agent http header. 
Currently we use these user agents:

1. "Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 
+http://sindice.com/developers/bot)"

2. "SindiceFetcher/Ping Manager (http://sindice.com/developers/bot"

3. "sindice.net ontology fetcher"

Niceness is implemented in our main fetcher. In some cases there may be 
bursts on sites providing distributed ontologies. Speaking with the 
group here it seems unlikely that we have not been hitting kaufkauf.net, 
  however if you can provide an IP address I can do some further 
verification.

I understand that http://lod.openlinksw.com/sparql is now hosted at 
DERI, and I wonder could some of the traffic be related to that? Again, 
if you can provide an IP address I will do some further verification.


Kind regards,
Rob.

--
Robert Fuller
Research Associate
DERI, Galway

Received on Tuesday, 8 June 2010 12:45:43 UTC