Re: Call for Concensus: Update SPARQL 1.1 test suite to align with changes introduced in RDF 1.1

On Nov 5, 2015, at 2:40 PM, Gregg Kellogg <gregg@greggkellogg.net> wrote:
> 
>> 1/ Where are the original tests the WG implementation report going to be?
>> 
>> There are links in the implementation report back to the tests under www.w3.org/2009/sparql/docs/tests/ so those would not be stable.
> 
> If we don’t change the semantics of any given test, that shouldn’t be a problem; the proposed changes should either simply mark the old tests as Obsolete, or create new tests, which are Proposed; this requires a change to the existing PR. An updated implementation report would either report values for obsoleted tests (if the implementation’s report wasn’t updated), or not have an entry for a new test.

As part of issue #20 ("Remove non-official tests from SPARQL 1.1 test suite”), I had hoped to clean up the SPARQL test suite so that it only contains current tests expected to be run as part of conformance checking. This means not only removing old tests that were never approved in the first place (or were approved and then simply dropped from the manifest lists), but also removing the obsoleted tests (like the ones I’ve just commented out in my PR in favor of the new RDF-1.1-compatible versions).

The deep structure of the test manifests is already rather complex and difficult for newcomers to understand. I think this would be made worse by a decision to keep unapproved, retracted, and obsoleted tests in the test suite.


(My PR #21 neither marks the old tests as obsolete, nor removes the tests. It simply removes the obsoleted tests from the manifest list. I had hoped this would very quickly be followed by a PR for #20 which would remove them entirely.)


> We need to weigh the plus/minus of keeping separate frozen snapshots of everything. That said, we can certainly create a frozen branch for the state of the test suite at any given point, and refer to that for any history that needs to be shown.

The “plus” here was that the SPARQL test suite was the result of the WG process.

> 
>> Or are all the links in the implementation report going to be updated to point to a frozen copy?
> 
> The implementation report used for transition is static and shouldn’t change. It should, however, contain a link to the most current implementation report, which can simply be the gh-pages version.

Are you suggesting we leave the implementation report frozen, but change the test suite in-place (at least in appearance based on the test suite URIs)? I would think both of these need to either stay frozen or be changed, but together.

thanks,
.greg

Received on Thursday, 5 November 2015 22:58:42 UTC