DAWG test suite & process overview

I've seen a few requests for an overview of the test setup from the 
first SPARQL working group (DAWG). I guess I know enough about it to 
volunteer.

SPARQL 1.0 tested 2 things: query & protocol.

== SPARQL 1.0 Query Testing ==

http://www.w3.org/2001/sw/DataAccess/tests/README.html gives the overall 
structure of the tests. To answer Axel's questions:

* How were the test cases collected in DAWG? Any naming conventions you 
followed?

==> The tests were grouped into directories within 
http://www.w3.org/2001/sw/DataAccess/tests/data-r2/ . There was no 
particular naming convention followed, other than the manifest file 
within each directory being named manifest.ttl

* Did anyone do some scripts to provide the manifests for testcases in 
DAWG or were these all assembled manually?

==> Manifests were assembled and maintained manually.

* Also, is it possible to have alternative results? (for 
non-deterministic queries, e.g. sample) in the test format of DAWG?

==> This was not supported in the test case vocabulary as is. The only 
SPARQL 1.0 feature that would need this would be REDUCED, I suppose. We 
handled this via (from the README):

"""
Query evaluation tests that involve the REDUCED keyword have slightly 
different passing criteria. These tests are indicated in the manifest 
files with the mf:resultCardinality predicate with an object of 
mf:LaxCardinality. To pass such a test, the result set produced by a 
SPARQL implementation must contain each solution in the expected result 
set at least once and no more than the number of times that the solution 
occurs in the expected result set. (That is, the expected result set 
contains the solutions with cardinalities as they would be if the query 
did not contain REDUCED; to pass the test, an implementation must 
produce the correct results with cardinalities between one and the 
cardainlity in the expected result set.)
"""

* Organisation of test cases: Shall we do separate manifests per 
feature? etc.

==> Yes, I think this makes sense.


I have a perl script lying around that generates this sort of overview 
document (http://www.w3.org/2001/sw/DataAccess/tests/r2) from the super 
manifest (manifest of manifests) and the individual manifests by doing a 
few SPARQL queries.

=== Results ===

We collected results via EARL as documented here:

http://www.w3.org/2001/sw/DataAccess/tests/earl

We did _not_ provide a harness to run the tests or generate results. 
Implementers provided their own results.

We had a tool chain courtesy of EricP that parsed the EARL and populated 
-- I think -- a mysql DB with the results, which was then used to 
generate the output reports at

   http://www.w3.org/2001/sw/DataAccess/impl-report-ql
   http://www.w3.org/2001/sw/DataAccess/tests/implementations

At one point in my life I think I knew how to use this tool chain, but 
I'll have to work to resuscitate that knowledge if we choose to use it. 
It relies on tests being associated with 'facets' being tested, in order 
to assign a score for how complete each implementation is for each facet.

== SPARQL 1.0 Protocol Testing ==

Elias Torres created a harness for performing the protocol tests. If I 
recall correctly, the tests were mainly based on the examples in the 
protocol specification. The tools are here:

http://www.w3.org/2001/sw/DataAccess/proto-tests/tools/

...but I don't remember much about how to use them. Basically, we 
pointed the tools at a SPARQL endpoint and it spit out result files 
which -- again, I think -- we manually compiled into the implementation 
report here:

http://www.w3.org/2001/sw/DataAccess/impl-report-protocol



...hope this is helpful...

Lee

Received on Thursday, 25 March 2010 04:34:23 UTC