Re: Call for Concensus: Update SPARQL 1.1 test suite to align with changes introduced in RDF 1.1

> On Nov 5, 2015, at 2:58 PM, Gregory Williams <greg@evilfunhouse.com> wrote:
> 
> On Nov 5, 2015, at 2:40 PM, Gregg Kellogg <gregg@greggkellogg.net> wrote:
>> 
>>> 1/ Where are the original tests the WG implementation report going to be?
>>> 
>>> There are links in the implementation report back to the tests under www.w3.org/2009/sparql/docs/tests/ so those would not be stable.
>> 
>> If we don’t change the semantics of any given test, that shouldn’t be a problem; the proposed changes should either simply mark the old tests as Obsolete, or create new tests, which are Proposed; this requires a change to the existing PR. An updated implementation report would either report values for obsoleted tests (if the implementation’s report wasn’t updated), or not have an entry for a new test.
> 
> As part of issue #20 ("Remove non-official tests from SPARQL 1.1 test suite”), I had hoped to clean up the SPARQL test suite so that it only contains current tests expected to be run as part of conformance checking. This means not only removing old tests that were never approved in the first place (or were approved and then simply dropped from the manifest lists), but also removing the obsoleted tests (like the ones I’ve just commented out in my PR in favor of the new RDF-1.1-compatible versions).
> 
> The deep structure of the test manifests is already rather complex and difficult for newcomers to understand. I think this would be made worse by a decision to keep unapproved, retracted, and obsoleted tests in the test suite.
> 
> 
> (My PR #21 neither marks the old tests as obsolete, nor removes the tests. It simply removes the obsoleted tests from the manifest list. I had hoped this would very quickly be followed by a PR for #20 which would remove them entirely.)

I think if we remove the tests from mf:entries, but keep the test definitions (and test files) with the approval status of Obsolete, that accomplishes what you want to do. Would that be okay? I don’t think we can or should completely eliminate history.

>> We need to weigh the plus/minus of keeping separate frozen snapshots of everything. That said, we can certainly create a frozen branch for the state of the test suite at any given point, and refer to that for any history that needs to be shown.
> 
> The “plus” here was that the SPARQL test suite was the result of the WG process.
> 
>> 
>>> Or are all the links in the implementation report going to be updated to point to a frozen copy?
>> 
>> The implementation report used for transition is static and shouldn’t change. It should, however, contain a link to the most current implementation report, which can simply be the gh-pages version.
> 
> Are you suggesting we leave the implementation report frozen, but change the test suite in-place (at least in appearance based on the test suite URIs)? I would think both of these need to either stay frozen or be changed, but together.

I was suggesting that we keep the implementation report stable, with a minor update to note the new location. This report was used for the CR transition, so it should remain unchanged for the historical record.

My opinion was that we could update the tests themselves, as long as references from the implementation report don’t change meaning. This is more friendly for developers going forward, and addresses some concerns about tests expecting to be run from a specific location. But, I can also see the value in keeping them frozen, given adequate notice about an update location.

I’m +1 for freezing the implementation report and updating the tests.
Otherwise, I’m -1 for updating the implementation report and +0.5 for freezing the existing tests with a reference to a new location.

Gregg

> thanks,
> .greg
>

Received on Thursday, 5 November 2015 23:07:33 UTC