TPF and DBMSes (was Re: Hydra and Shapes)

On Friday 21. November 2014 13.44.02 Kingsley Idehen wrote:
> To be more precise, since I don't like DBMS and Database Document 
> conflation either, A SPARQL processor needs a mechanism for targeting 
> database documents.
> 
> In the case of 
> <http://client.linkeddatafragments.org/#sparql-processor>, how does 
> handle what I've outlined above work?

As they said, it is just the UI, you can hand any TPF data source to the client and off you go.

But I think that misses the crucial point, which is how things happen behind that. Strictly, Ruben is right; you 
can basically materialize all possible triple patterns, with pages, and store them in a file system. In that 
case, it is correct that no DBMS is involved. 

However, I would claim that this is not practical in almost all cases. I have myself been working on a system 
that did something similar, in a fairly high-traffic, but slow-update case, which is the case where it possibly 
makes most sense, but the horror of the code we had to manage the updates, I still shiver at the thought! 
:-) Essentially, it is writing YA materialization framework.

I think you'd much rather like to ensure that you have a good DBMS at the bottom, then a layer that serves 
TPFs, and makes sure you have cache headers, and then a reverse caching proxy facing the network 
(which is about the architecture that I'm running on my endpoint).

However, this places three requirements that aren't necessarily easy on the DBMS. The first, that I know 
that you do well, already, Kingsley, is paging. The next is that it must be much cheaper to compute the 
cardinality (or at least an estimate) of any triple pattern result, than computing the result of the same triple 
pattern. Can you do that with Virtuoso? If yes, how do we access it?

The third requirement is that it must be much cheaper to compute the last modified time of any triple 
pattern result than to compute the result itself, so that the reverse proxy can do its job effectively. Again, 
can Virtuoso do that, and if so, how can we access that time?


Best,

Kjetil