Re: DAWG Agenda - 28 Aug 2007 @ 14:30 UTC

Eric Prud'hommeaux wrote:
> * Lee Feigenbaum <lee@thefigtrees.net> [2007-08-27 15:04-0400]
>> 5. Test suite and test service and implementation report
>>
>> We've published the test suite and will be publishing the test service 
>> shortly. Where should we gather EARL results? Any progress on generating an 
>> implementation report?
> 
> Quick breakdown of the features passed vs. the features *tested*.
> (ARQ's report included every test.)
> +-----------------------------------------------------------------------------+-----------+----------+
> | RDF API for PHP                                                             | SUM(pass) | COUNT(*) |
> +-----------------------------------------------------------------------------+-----------+----------+
> | ASK query form                                                              |         0 |        9 |
> | Basic graph pattern matching. Triple pattern constructs. Blank node scoping |        64 |      171 |
> | Core bits of SPARQL. Prefixed names, variables, blank nodes, graph terms    |       316 |      845 |
> | RDF datasets. Default and named graphs. GRAPH keyword                       |         0 |        7 |
> | FILTER clauses and expressions                                              |        76 |      232 |
> | OPTIONAL pattern matching                                                   |         5 |       19 |
> | SELECT query form                                                           |        64 |      167 |
> | Sorting (ORDER BY) and slicing (LIMIT, OFFSET)                              |        13 |       26 |
> | UNION pattern matching                                                      |         1 |        4 |
> +-----------------------------------------------------------------------------+-----------+----------+
> | ARQ                                                                         | SUM(pass) | COUNT(*) |
> +-----------------------------------------------------------------------------+-----------+----------+
> | ASK query form                                                              |       102 |      102 |
> | Basic graph pattern matching. Triple pattern constructs. Blank node scoping |       303 |      303 |
> | CONSTRUCT query form                                                        |        15 |       15 |
> | Core bits of SPARQL. Prefixed names, variables, blank nodes, graph terms    |      1353 |     1353 |
> | RDF datasets. Default and named graphs. GRAPH keyword                       |        44 |       44 |
> | FILTER clauses and expressions                                              |       545 |      545 |
> | OPTIONAL pattern matching                                                   |        25 |       25 |
> | SELECT query form                                                           |       236 |      236 |
> | Sorting (ORDER BY) and slicing (LIMIT, OFFSET)                              |        92 |       92 |
> | UNION pattern matching                                                      |        12 |       12 |
> +-----------------------------------------------------------------------------+-----------+----------+
> 
> for a more tedious picture, we can look at RDF API for PHP by facet:
[snipped]

> I'd prefer if the passed numbers were either 0 or equal to the sum
> (i.e. we had some confidence that we were isolating the parts of their
> code that followed the spec or not). We could do some forensics, but I
> think we should wait 'till they re-run now that we have a full test
> set for them.

I agree with this preference, but am not sure how close we are to 
achieving it. Do the pass numbers you're showing here include having 
applied the canonical-test analysis yet? I think that the real goal 
should be that once that heuristic is applied then an implementation 
either passes or fails all tests for a given facet. (I am pessimistic 
that we will achieve that goal however.)

> I did $(wget)s into a tests/reports/ directory and modified the
> reports to make them uniform. I separated ARQ's assertor and
> subject. RDF API for PHP had 19 mislabled tests so I commented those
> out. Likewise, I commented theirs that dealt with the syntax
> tests. Andy nicely separated his syntax from his semantics tests.
> 
> RDF API for PHP used a structure like this to describe the earl:subject:
> <http://rdfapi-php.sf.net/> doap:name "RDF API for PHP";
>     doap:release [ doap:created "2007-08-13 17:09"^^<http://www.w3.org/2001/XMLSchema#date>;
>         doap:name "RAP SVN-2007-08-13T17:09";
>         a doap:Version ];
>     a doap:Project.
> so I emulated that in ARQ's report:
> _:b1
>       rdf:type      earl:Software ;
>       doap:name     "ARQ" ;
>       doap:release  [ doap:name "@version@" ] ;
>       <http://purl.org/dc/terms/isPartOf>
>                     [ foaf:homepage  <http://jena.sf.net/>
>                     ] ;
>       foaf:homepage  <http://jena.sf.net/ARQ> .
> (The "@version@" had been on a dc:somethingorother arc.)
> 
> Should I check in the reports/ dir?

Please do.

Lee

Received on Tuesday, 28 August 2007 14:16:40 UTC