Re: SPARQL test suite as a Docker container?

On Sat, May 25, 2024 at 12:15 AM Gregory Williams <greg@evilfunhouse.com> wrote:
>
> On May 24, 2024, at 1:46 PM, Martynas Jusevičius <martynas@atomgraph.com> wrote:
> >
> > Hi,
> >
> > I was trying to figure out how to run the SPARQL test suite which lead
> > me to links
> > https://github.com/w3c/rdf-tests/issues/47
> > https://github.com/kasei/sparql11-protocolvalidator
> > and several dead links too.
> >
> > I don't have the answer yet. It doesn't look good for a standards suite IMO.
>
> Those links are specifically about testing SPARQL Protocol implementations. There's probably similar variation in how people run the tests for query, update, etc. The protocol validator you linked to is mine, and was developed during the SPARQL 1.1 standardization process to support the working group’s need to evaluate the implementation support for the SPARQL Protocol. I’m not opposed to it being used on an ongoing basis, but it’s not actively being maintained. I’d be happy to accept pull requests for adding a Dockerfile if anyone wants to attempt that.

And how do we run the query test suite, not the protocol suite?

> > Moreover, I found a quote "everyone has their own mechanism for
> > running the tests" (I suppose meaning every implementation), which is
> > weird given that SPARQL provides a standard protocol for running
> > queries.
>
> I think mostly the “everyone has their own mechanism” situation is because to do the testing using just standard APIs, you’d have to assume that part of the system under test already worked (e.g. assuming SPARQL Update or GSP worked so that you could set up the dataset to run a query test).

I'm willing to assume that the system worked if that means I can have
a single test suite that works with all RDF triplestores. Nobody can
figure out all the vendor-specific ways to run tests and keep them
up-to-date. Most test suite reports are a decade old.

I think having a compatibility coverage overview of different products
would be good for the industry. Having an easy to run test suite is a
prerequisite of that IMO.

http://sparqlscore.com seems to be an attempt at something similar but
it looks like it's not up to date, which just proves my point.

> The Protocol tests are a bit different in this respect, because they build on top of the other specs, and the GitHub repo states right up front in the README that full support of SPARQL 1.1 Query and Update is assumed.
>
> .greg
>

Received on Saturday, 25 May 2024 20:17:58 UTC