toward an implementation report - an update

EricP and I did some work on how we'll present an implementation report.

The basic idea is to define a relatively small and manageable number of 
high-level SPARQL features. (Our CR exit criteria mandate two 
interoperable implementations of each of these features.) Additionally, 
we need to define what it means for an implementation to pass each of 
these features.

Here's what Eric and I are suggesting for now:

+ Every test has N facets, for some N >= 0. (In practice, >= 5 or so.)
+ Every facet belongs to 1 feature. (Every feature has M facets, M >= 1.)
+ Every facet has one canonical test. This is basically a test that 
tests as little as possible beyond that facet. (Right now, the canonical 
tests are determined automatically by Eric's facets-test.pl. We're going 
to eyeball this to make sure it seems reasonable.)
+ When we examine an implementation's EARL results, every failing test 
can count against 1 or more facets:
   + Failing test T that has facets f1, f2, ..., fN will count against 
facet fi if the implementation failed the canonical test for fi. (This 
rule tries to explain the cause of the failure and avoid blaming other 
parts. If I fail SELECT * { A OPTIONAL B } but pass the canonical test 
for select-*, failing this test shouldn't count against the select-* facet.)
   + Every failing test must count against at least one facet - if the 
above rule doesn't find an explanation, then manual intervention 
examines the case and picks a most-likely-responsible facet (or, if 
appropriate, creates a new facet) to count against.
   + The test counts as a failure against each feature that contains a 
facet that the test counts against.

So at that point, we can look at EARL results and come up with a split 
on (explicitly) passed/failed tests for each feature.

We'll also provide the detailed breakdown by test and by facet of 
implementation results.

For now, I've identified the following high-level features:

:CoreSPARQL a test:DAWGfeature ;
   rdfs:comment "Core bits of SPARQL. Prefixed names, variables, blank 
nodes, graph terms" .

:BasicGraphPatterns a test:DAWGfeature ;
   rdfs:comment "Basic graph pattern matching. Triple pattern 
constructs. Blank node scoping" .

:Optionals a test:DAWGfeature ;
   rdfs:comment "OPTIONAL pattern matching" .

:Unions a test:DAWGfeature ;
   rdfs:comment "UNION pattern matching" .

:Filters a test:DAWGfeature ;
   rdfs:comment "FILTER clauses and expressions" .

:Datasets a test:DAWGfeature ;
   rdfs:comment "RDF datasets. Default and named graphs. GRAPH keyword" .

:SolutionSequence a test:DAWGfeature ;
   rdfs:comment "Sorting (ORDER BY) and slicing (LIMIT, OFFSET)" .

:SelectQueries a test:DAWGfeature ;
   rdfs:comment "SELECT query form" .

:ConstructQueries a test:DAWGfeature ;
   rdfs:comment "CONSTRUCT query form" .

:AskQueries a test:DAWGfeature ;
   rdfs:comment "ASK query form" .

It's unclear to me whether "syntax", "type promotion", "i18n" should be 
their own features, or whether anything else should be. I think this is 
a decent list to start, and I've gone ahead and updated facets.ttl to 
put each facet in the proper feature.

In annotating the facets, I noticed that we don't seem to have any 
facets that capture the following types of tests (we do _have_ tests for 
these, but I think right now they won't look special in the results):

   - internationalization
   - type promotion
   - blank node label scoping
   - FILTER scoping

We can leave those alone and see if a pattern emerges in the 
implementation results, or we can try to identify facets for them (XPath 
-based if possible, and otherwise manual/other - EricP?)


thoughts?
Lee

Received on Friday, 3 August 2007 14:35:30 UTC