Re: Fwd: Light-weight streaming PHP library for RDF serialization?

Nicholas,
 Bio2RDF scripts [1] process mixed media - flat files, tab files, xml files
- into RDF [2], and this is coordinated with a common API [3]. Some of
these files are rather large, so it's important that we occasionally
serialize the contents otherwise we'll run out of space. Since in only very
rare times do we actually want to query the model, we don't need it in
memory, but just keep an simple index if necessary. We also need support
for generating n-quads.  Although I've been thinking of redeveloping my api
with easyRDF (while also taking advantage of composer for dependency
management and sami for documentation), streaming and nquad support are
essential requirements.

m.

[1] https://github.com/bio2rdf/bio2rdf-scripts
[2] https://github.com/bio2rdf/bio2rdf-scripts/wiki
[3] https://github.com/micheldumontier/php-lib




On Tue, Jan 29, 2013 at 3:42 PM, Nicholas J Humfrey <njh@aelius.com> wrote:

> Hello Denny,
>
> Sorry, I am not on the semantic-web mailing list - too many emails for me
> to take in and not enough time. Stéphane kindly forwarded your email to
> me.
>
>
> No, it is not currently possible to serialise a triple stream with EasyRdf.
> I took this decision for a number of reasons:
>
> 1) EasyRdf was designed with the BBC's web platform in mind. This
> typically uses Java (and others) as a 'heavy lifting' service layer and
> PHP as a lightweight presentation layer. As such PHP should only have a
> single page worth of data to process at a time - thus streaming was not an
> important requirement.
>
> 2) At the core of the EasyRdf is the a graph model object. EasyRdf started
> off as an object model layer on top of ARC2 (and others). Since ARC2 has
> had less development work done on it, I have been expanding the number of
> native parsers and serialisers in it. I want to avoid making it overly
> complex with multiple APIs for doing similar things (!)
>
> 3) The HTTP client API that I have been using (based on Zend_HTTP_Client,
> which is again what the BBC uses) doesn't support streaming - it loads the
> full response into memory. Therefore there are fewer benefits in EasyRdf
> being able to stream triples.
>
> 4) I have worked hard to try and make the RDF/XML and Turtle
> serialisations as pretty as possible - this involves collecting/sorting
> all the same resources and properties together, so that the document reads
> well. Otherwise you just end up with a triple oriented document that reads
> like N-Triples or Trix. Some implementations (such as Redland) do this
> within the serialiser itself but that seemed like an extra overhead, when
> I already had the data organised like that inside the EasyRdf graph
> object.
>
>
> Having said all of that, some of the serialisers would be fairly easy to
> convert and I would be willing to look at changing the API in order to
> help you with your requirements (I am a big fan of WikiData!). It would
> also make sense to not have multiple PHP libraries for serialising RDF,
> with varying quality and features - I think this is one of the reasons why
> the semantic web hasn't taken off faster.
>
>
> What is your streaming source of triples?
> Are you serialising direct from the database?
> Can the database pre-sort subjects and properties, so they are ready to be
> serialised?
> Is this for a bulk-export or individual API queries?
>
>
> nick.
>
>
> > ---------- Forwarded message ----------
> > From: Denny Vrandečić <denny.vrandecic@wikimedia.de>
> > Date: Tue, Jan 29, 2013 at 11:54 AM
> > Subject: Light-weight streaming PHP library for RDF serialization?
> > To: SW-forum <semantic-web@w3.org>
> >
> >
> > Hi,
> >
> > is there an actively maintained open source pure PHP library that can be
> > used to create RDF serialization from a model?
> >
> > It should be able to stream a big number of triples.
> >
> > Pluspoints if there it has no Parser or SPARQL processing library as a
> > dependency, in order to decrease the size of the library (smaller library
> > =
> > happier code reviewer, less maintenance costs).
> >
> > Cheers,
> > Denny
> >
> > --
> > Project director Wikidata
> > Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin
> > Tel. +49-30-219 158 26-0 | http://wikimedia.de
> >
> > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
> > Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> > unter
> > der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> > Körperschaften I Berlin, Steuernummer 27/681/51985.
> >
> >
> >
> > --
> > Steph.
> >
>
>
>
>


-- 
Michel Dumontier
Associate Professor of Bioinformatics, Carleton University
Chair, W3C Semantic Web for Health Care and the Life Sciences Interest Group
http://dumontierlab.com