Re: RDF/JSON

Thanks, Sandro, for sorting the admin issues.  I've moved the cc on this message to public-rdf-comments in case Martin replies.

Martin, are you not conflating data formats with data models?  Almost all current development patterns rely upon the technical components agreeing to a data format.  Thus, you are encouraged to have a client absorb the data data format used by your Mongo or Lucene DBs.  The thing with RDF is that we have many formats that use the same data model.  Why not convert between formats that best suit processing at any given stage?

Regards,
Dave


On Apr 26, 2013, at 23:17, Sandro Hawke <sandro@w3.org> wrote:

> On 04/25/2013 12:11 PM, Arnaud Le Hors wrote:
>> My colleague Martin Nally sent a response to Markus but because he's not subscribed to the list his message was put on hold for moderation. I thought this would only take a day or two but it's not happened so I'm forwarding his message to avoid any further delay. I suggest you copy him in any response though. 
>> Thanks.
> 
> 
> Sorry about this - only WG members can post to the WG list; everyone else is supposed to post to public-rdf-comments.  That's a bit of pain, but (ironically) it's supposed to help us make sure we're not forgetting to reply to any comments.   The main failing is the reply that's sent says something else; I've put in a request to change that.
> 
> On the substance of the comment -- I find it rather compelling and am not sure how to proceed.   The bit about "Databases like to query on predicates" was particularly interesting....   An odd contrast to direct access from JS.
> 
> Arnaud/Martin, how ideally would you like to proceed on this?   Just make RDF/JSON a note?   Or something else?    Should the two be aligned in those "details" where they differ?  On the point about documentation, would it help if there were a simple, mostly self-contained section of the JSON-LD spec you could point to?
> 
>      -- Sandro 
> 
>> --
>> Arnaud  Le Hors - Software Standards Architect - IBM Software Group
>> 
>> ----- Forwarded by Arnaud Le Hors/Cupertino/IBM on 04/25/2013 09:01 AM ----- 
>> 
>> From:        Martin Nally <martin.nally@gmail.com> 
>> To:        Arnaud Le Hors <lehors@us.ibm.com> 
>> Date:        04/17/2013 12:21 PM 
>> Subject:        Fwd: RDF/JSON 
>> 
>> 
>> Markus, 
>> 
>> Thanks for your careful response to my code samples trying to explain why RDF/JSON has been working better for us than JSON-LD. Let me first give some background. The applications we have been writing have logic-tier servers written in python (and some ruby) with user-interfaces written in javascript running in the browser on PCs and mobile devices. The JSON we are discussing is the JSON that flows between the python/ruby and the javascript. Both sides are RDF-aware, and many of our resources have multiple subjects. The JSON forms the primary API of the servers, although you can also ask for the same data in turtle or rdfa (we use the rdfa format for some use-cases in our applications, but we are not currently using turtle). We do not have a very long or broad experience - we are a prototyping team, not (yet) a product team, and we have built up no more than a few of thousand lines of code in these applications.
>> 
>> Our JSON was originally in JSON-LD format, in the sense that the JSON we produced and consumed was valid JSON-LD, but we only produced and consumed a very restricted subset of JSON-LD. For example, we did not support contexts. What we used from JSON-LD was the basic organization that can be perhaps be summarized like this: [{'@id': S, P: O}]. I hope that is clear - it is an array of "dictionaries" where one dictionary entry is '@id' for the subject, and the other dictionary entries provide the predicates and objects. The O is itself an array of dictionaries, with the only valid keys in the dictionaries being '@id', '@value' and '@type'.
>> 
>> We spent a few months building code this way, and it worked OK. We also stored this format in MongoDB and queried on it successfully. I cannot remember exactly what triggered the decision, but about 2 months ago, I decided to convert over to RDF/JSON format. Since RDF/JSON has almost no options - in contrast with JSON-LD, which has many - you don't need me to explain what we did much further, but for completeness I will say simply that we moved to this data organization: {S: { P: O}}. The O in RDF/JSON is very similar to the JSON-LD version of O we used before - it differs only in the detail.
>> 
>> My experience with the port is that the complexity of our code came down substantially. I would guess the code that manipulates RDF is now in general maybe only two thirds or three-quarters of what it had previously been. Even where the complexity is similar, we now have the advantage of using simpler language constructs (primarily dictionary access like a[b]), where before we used a helper method. Not all our helper methods are gone - for example we still have a helper method whose meaning is "for a given predicate and object, return all the subjects". It is possible that the simplification we experienced was particular to our data. I think you are correct in saying that if our representations were all single-subject, then we would not have found RDF/JSON to be better - the JSON-LD data organization would have been as good or better. Other than the fact that our resources often have multiple subjects, I doubt there is anything very special in what we are doing. I had hoped that my code samples might help explain why our code got simpler, but I can see now that probably won't work.
>> 
>> Interestingly, RDF/JSON turned out not to be helpful at all in JSON databases. Databases like to query on predicates, and JSON-LD's approach of making the subject be the value of the '@id' predicate has worked much better for us, so our database format is still a very restricted but valid JSON-LD format. (We have also used triples stores - that is another conversation.) We have gone back and forth on whether we prefer         JSON-LD's format for the 'O' part or RDF/JSON's version. We have seen advantages to both. When we put our JSON into Lucene, JSON-LD's version of the O format was more convenient, because it allowed us to easily tell Lucene to handle URLs differently from regular strings. When we put the data into MongoDB, we did not need this, and RDF/JSON's version of the O part looked better because it did not require query writers to remember to use '@ID' for URL-valued properties and '@value' for everything else - you just always use 'value'. This is particularly helpful if you don't know what type you are querying on. Right now, we are using both 'O' formats, one for MongoDB and the other for Lucene.
>> 
>> In addition to the code simplification we saw when we converted to RDF/JSON for the API, we also saw a reduction in specification complexity that may ultimately be more important. Remember that our JSON is our API and so we need to document it. One approach is to document it ourselves without reference to any external specification whatever. In that case, documenting [{'@id': S, P: O}] is only slightly more complex than documenting {S: { P: O}}. However, we would really rather not document this ourselves - we would rather point to a specification. For {S: { P: O}} we can easily point to the RDF/JSON spec - the whole spec almost fits on a page, and since RDF/JSON really has no options, there is little to say about our usage patterns of it. By contrast, JSON-LD is a complex specification with many options. If we were to reference the JSON-LD spec in our API documentation, not only would we be referencing a relatively large and complex spec, we would then have to add yet more information to document the very restricted subset we actually use. We get no value from the parts of JSON-LD we do not use and we have no interest in allowing         clients to give us arbitrary JSON-LD, or in producing it, either now or in the future. Referencing the JSON-LD spec in our API documentation is not an attractive option.
>> 
>> I hope this helps explain our usage of RDF/JSON and JSON-LD and our experiences with them.
>> 
>> Best wishes, Martin 
>