- From: Lee Feigenbaum <lee@thefigtrees.net>
- Date: Tue, 28 Aug 2007 09:54:14 -0400
- To: Eric Prud'hommeaux <eric@w3.org>
- CC: 'RDF Data Access Working Group' <public-rdf-dawg@w3.org>
Eric Prud'hommeaux wrote: > * Lee Feigenbaum <lee@thefigtrees.net> [2007-08-27 15:04-0400] >> 5. Test suite and test service and implementation report >> >> We've published the test suite and will be publishing the test service >> shortly. Where should we gather EARL results? Any progress on generating an >> implementation report? > > Quick breakdown of the features passed vs. the features *tested*. > (ARQ's report included every test.) > +-----------------------------------------------------------------------------+-----------+----------+ > | RDF API for PHP | SUM(pass) | COUNT(*) | > +-----------------------------------------------------------------------------+-----------+----------+ > | ASK query form | 0 | 9 | > | Basic graph pattern matching. Triple pattern constructs. Blank node scoping | 64 | 171 | > | Core bits of SPARQL. Prefixed names, variables, blank nodes, graph terms | 316 | 845 | > | RDF datasets. Default and named graphs. GRAPH keyword | 0 | 7 | > | FILTER clauses and expressions | 76 | 232 | > | OPTIONAL pattern matching | 5 | 19 | > | SELECT query form | 64 | 167 | > | Sorting (ORDER BY) and slicing (LIMIT, OFFSET) | 13 | 26 | > | UNION pattern matching | 1 | 4 | > +-----------------------------------------------------------------------------+-----------+----------+ > | ARQ | SUM(pass) | COUNT(*) | > +-----------------------------------------------------------------------------+-----------+----------+ > | ASK query form | 102 | 102 | > | Basic graph pattern matching. Triple pattern constructs. Blank node scoping | 303 | 303 | > | CONSTRUCT query form | 15 | 15 | > | Core bits of SPARQL. Prefixed names, variables, blank nodes, graph terms | 1353 | 1353 | > | RDF datasets. Default and named graphs. GRAPH keyword | 44 | 44 | > | FILTER clauses and expressions | 545 | 545 | > | OPTIONAL pattern matching | 25 | 25 | > | SELECT query form | 236 | 236 | > | Sorting (ORDER BY) and slicing (LIMIT, OFFSET) | 92 | 92 | > | UNION pattern matching | 12 | 12 | > +-----------------------------------------------------------------------------+-----------+----------+ > > for a more tedious picture, we can look at RDF API for PHP by facet: [snipped] > I'd prefer if the passed numbers were either 0 or equal to the sum > (i.e. we had some confidence that we were isolating the parts of their > code that followed the spec or not). We could do some forensics, but I > think we should wait 'till they re-run now that we have a full test > set for them. I agree with this preference, but am not sure how close we are to achieving it. Do the pass numbers you're showing here include having applied the canonical-test analysis yet? I think that the real goal should be that once that heuristic is applied then an implementation either passes or fails all tests for a given facet. (I am pessimistic that we will achieve that goal however.) > I did $(wget)s into a tests/reports/ directory and modified the > reports to make them uniform. I separated ARQ's assertor and > subject. RDF API for PHP had 19 mislabled tests so I commented those > out. Likewise, I commented theirs that dealt with the syntax > tests. Andy nicely separated his syntax from his semantics tests. > > RDF API for PHP used a structure like this to describe the earl:subject: > <http://rdfapi-php.sf.net/> doap:name "RDF API for PHP"; > doap:release [ doap:created "2007-08-13 17:09"^^<http://www.w3.org/2001/XMLSchema#date>; > doap:name "RAP SVN-2007-08-13T17:09"; > a doap:Version ]; > a doap:Project. > so I emulated that in ARQ's report: > _:b1 > rdf:type earl:Software ; > doap:name "ARQ" ; > doap:release [ doap:name "@version@" ] ; > <http://purl.org/dc/terms/isPartOf> > [ foaf:homepage <http://jena.sf.net/> > ] ; > foaf:homepage <http://jena.sf.net/ARQ> . > (The "@version@" had been on a dc:somethingorother arc.) > > Should I check in the reports/ dir? Please do. Lee
Received on Tuesday, 28 August 2007 14:16:40 UTC