W3C home > Mailing lists > Public > public-lod@w3.org > June 2011

Re: Think before you write Semantic Web crawlers

From: Richard Cyganiak <richard@cyganiak.de>
Date: Wed, 22 Jun 2011 22:49:36 +0100
Cc: public-lod@w3.org
Message-Id: <A8E03EC5-99D7-4C60-A20D-68A9B60A2DFA@cyganiak.de>
To: Martin Hepp <martin.hepp@ebusiness-unibw.org> 
On 21 Jun 2011, at 10:44, Martin Hepp wrote:
> PS: I will not release the IP ranges from which the trouble originated, but rest assured, there were top research institutions among them.

The right answer is: name and shame. That is the way to teach them.

Like Karl said, we should collect information about abusive crawlers so that site operators can defend themselves. It won't be *that* hard to research and collect the IP ranges of offending universities.

I started a list here:
http://www.w3.org/wiki/Bad_Crawlers

The list is currently empty. I hope it stays that way.

Thank you all,
Richard
Received on Wednesday, 22 June 2011 21:50:06 UTC

This archive was generated by hypermail 2.4.0 : Thursday, 24 March 2022 20:29:54 UTC