Re: Think before you write Semantic Web crawlers

On Wed, Jun 22, 2011 at 7:08 PM, Sebastian Schaffert <
sebastian.schaffert@salzburgresearch.at> wrote:

>
> Am 22.06.2011 um 23:01 schrieb Lin Clark:
>
> > On Wed, Jun 22, 2011 at 9:33 PM, Sebastian Schaffert <
> sebastian.schaffert@salzburgresearch.at> wrote:
> >
> > Your complaint sounds to me a bit like "help, too many clients access my
> data".
> >
> > I'm sure that Martin is really tired of saying this, so I will reiterate
> for him: It wasn't his data, they weren't his servers. He's speaking on
> behalf of people who aren't part of our insular community... people who
> don't have a compelling reason to subsidize a PhD student's Best Paper award
> with their own dollars and bandwidth.
>
> And what about those companies subsidizing PhD students who write crawlers
> for the normal Web? Like Larry Page in 1998?
>
>
Talking to some friends at Stanford, they told me some of the problems that
they would go through initially. For example, many website owners would
directly call Stanford and threaten to sue them because of google's
crawlers.




> >
> > Agents can use Linked Data just fine without firing 150 requests per
> second at a server. There are TONs of use cases that do not require that
> kind of server load.
>
> And what if in the future 100.000 software agents will access servers? We
> will have the scalability issue eventually even without crawlers, so let's
> try to solve it. In the eyeball web, there are also crawlers without too
> much of a problem, and if Linked Data is to be successful we need to do the
> same.
>
> Greetings,
>
> Sebastian
> --
> | Dr. Sebastian Schaffert          sebastian.schaffert@salzburgresearch.at
> | Salzburg Research Forschungsgesellschaft  http://www.salzburgresearch.at
> | Head of Knowledge and Media Technologies Group          +43 662 2288 423
> | Jakob-Haringer Strasse 5/II
> | A-5020 Salzburg
>
>
>

Received on Wednesday, 22 June 2011 23:59:55 UTC