W3C home > Mailing lists > Public > public-lod@w3.org > April 2010

Re: DBpedia hosting burden

From: Ivan Mikhailov <imikhailov@openlinksw.com>
Date: Thu, 15 Apr 2010 01:49:23 +0700
To: Dan Brickley <danbri@danbri.org>
Cc: Kingsley Idehen <kidehen@openlinksw.com>, public-lod <public-lod@w3.org>, dbpedia-discussion <dbpedia-discussion@lists.sourceforge.net>
Message-Id: <1271270963.3114.946.camel@octo.iv.dev.null>
Dan,

> Are there any scenarios around eg. BitTorrent that could be explored?
> What if each of the static files in http://dbpedia.org/sitemap.xml
> were available as torrents (or magnet: URIs)? I realise that would
> only address part of the problem/cost, but it's a widely used
> technology for distributing large files; can we bend it to our needs?

If I were The Emperor of LOD I'd ask all grand dukes of datasources to
put fresh dumps at some torrent with control of UL/DL ratio :) For
reason I can't understand this idea is proposed few times per year but
never tried.

Other approach is to implement scalable and safe patch/diff on RDF
graphs plus subscription on them. That's what I'm writing ATM. Using
this toolkit, it would be quite cheap to place a local copy of LOD on
any appropriate box in any workgroup. A local copy will not require any
hi-end equipment for two reasons: the database can be much smaller than
the public one (one may install only a subset of LOD) and it will
usually less sensitive to RAM/disk ratio (small number of clients will
result in better locality because any given individual tend to browse
interrelated data whereas a crowd produces chaotic sequence of
requests). Crawlers and mobile apps will not migrate to local copies,
but some complicated queries will go away from the bottleneck server and
that would be good enough.

Best Regards,

Ivan Mikhailov
OpenLink Software
http://virtuoso.openlinksw.com
Received on Wednesday, 14 April 2010 18:49:55 UTC

This archive was generated by hypermail 2.3.1 : Sunday, 31 March 2013 14:24:26 UTC