- From: Lee Feigenbaum <lee@thefigtrees.net>
- Date: Thu, 25 Mar 2010 00:33:45 -0400
- To: SPARQL Working Group <public-rdf-dawg@w3.org>
I've seen a few requests for an overview of the test setup from the first SPARQL working group (DAWG). I guess I know enough about it to volunteer. SPARQL 1.0 tested 2 things: query & protocol. == SPARQL 1.0 Query Testing == http://www.w3.org/2001/sw/DataAccess/tests/README.html gives the overall structure of the tests. To answer Axel's questions: * How were the test cases collected in DAWG? Any naming conventions you followed? ==> The tests were grouped into directories within http://www.w3.org/2001/sw/DataAccess/tests/data-r2/ . There was no particular naming convention followed, other than the manifest file within each directory being named manifest.ttl * Did anyone do some scripts to provide the manifests for testcases in DAWG or were these all assembled manually? ==> Manifests were assembled and maintained manually. * Also, is it possible to have alternative results? (for non-deterministic queries, e.g. sample) in the test format of DAWG? ==> This was not supported in the test case vocabulary as is. The only SPARQL 1.0 feature that would need this would be REDUCED, I suppose. We handled this via (from the README): """ Query evaluation tests that involve the REDUCED keyword have slightly different passing criteria. These tests are indicated in the manifest files with the mf:resultCardinality predicate with an object of mf:LaxCardinality. To pass such a test, the result set produced by a SPARQL implementation must contain each solution in the expected result set at least once and no more than the number of times that the solution occurs in the expected result set. (That is, the expected result set contains the solutions with cardinalities as they would be if the query did not contain REDUCED; to pass the test, an implementation must produce the correct results with cardinalities between one and the cardainlity in the expected result set.) """ * Organisation of test cases: Shall we do separate manifests per feature? etc. ==> Yes, I think this makes sense. I have a perl script lying around that generates this sort of overview document (http://www.w3.org/2001/sw/DataAccess/tests/r2) from the super manifest (manifest of manifests) and the individual manifests by doing a few SPARQL queries. === Results === We collected results via EARL as documented here: http://www.w3.org/2001/sw/DataAccess/tests/earl We did _not_ provide a harness to run the tests or generate results. Implementers provided their own results. We had a tool chain courtesy of EricP that parsed the EARL and populated -- I think -- a mysql DB with the results, which was then used to generate the output reports at http://www.w3.org/2001/sw/DataAccess/impl-report-ql http://www.w3.org/2001/sw/DataAccess/tests/implementations At one point in my life I think I knew how to use this tool chain, but I'll have to work to resuscitate that knowledge if we choose to use it. It relies on tests being associated with 'facets' being tested, in order to assign a score for how complete each implementation is for each facet. == SPARQL 1.0 Protocol Testing == Elias Torres created a harness for performing the protocol tests. If I recall correctly, the tests were mainly based on the examples in the protocol specification. The tools are here: http://www.w3.org/2001/sw/DataAccess/proto-tests/tools/ ...but I don't remember much about how to use them. Basically, we pointed the tools at a SPARQL endpoint and it spit out result files which -- again, I think -- we manually compiled into the implementation report here: http://www.w3.org/2001/sw/DataAccess/impl-report-protocol ...hope this is helpful... Lee
Received on Thursday, 25 March 2010 04:34:23 UTC