Re: formal objection to early cutoff of test submissions

Test cases often arise from implementation experience. The candidate
recommendation period is designed to provide a period when potential
implementers can produce implementations.  The candidate recommendation
period now also serves as a last-call period, the period when many
interested parties first look at a potential recommendation and thus may
submit test cases.

For SHACL the candidate recommendation period was advertised as a four-week
period, from 11 April to 9 May.  The initial cut-off date for test suite
submissions was 25 April, cutting of one-half of the advertised period when
test case submissions are to be expected.

W3C is supposed to welcome comments on its potential recommendations, and so
W3C working groups should be committed to examining problems that are
discovered with its potential recommendations all the way up to them
becoming official W3C recommendations, and even afterwards.

Instead of cutting off test suite submissions so early, the working group
should be accepting test suite submissions all the way up to the transition
to proposed recommendation, and even after.  Certainly late submissions
might have to provide signifficant new information to be incorporated in the
test suite.

It appears the the working group has changed its policy and will now
continue to accept test suite submissions subject to a vote of the working
group.  If the policy on test suite submissions has changed and this change
in policy is announced, this formal objection is moot.

Peter F. Patel-Schneider
Nuance Communications


On 05/02/2017 09:58 AM, Irene Polikoff wrote:
> To me, this sounds like a procedural objection:
> 
>>> Cutting off the submission of tests well before the
>>> end of the candidate recommendation period hinders wide review of the
>>> candidate recommendation and goes against the purpose of the W3C process.
> 
> I think the key phrase is "well before the end of the candidate recommendation
> period”. 
> 
> I believe the process has been followed:
> 
>   * Obviously, submission of tests to be used as part of the recommendation
>     track process needs to be concluded before the end of the end of the
>     candidate recommendation period.
>       o This is because implementers need time to run all tests and make fixes
>         if necessary.
>       o Then the WG needs to review the results, draft the transition request
>         and decide to submit it.
>       o Then request to transition has to be sent and transition meeting has
>         to happen. 
>       o All these activities have to occur before the candidate recommendation
>         period ends. Conservatively speaking, they take at least a month.
>   * Further, the WG issued a “last call” for tests asking any interested
>     parties to let us know if they need extra time and, if so, how much. We
>     promised to “work something out” if such parties tell us about their plans
>     for submitting
>     tests https://lists.w3.org/Archives/Public/public-rdf-shapes/2017Apr/0034.html
>       o The last call also said that additional tests can be submitted at any
>         time - as something for the implementers to test with, but, after the
>         deadline, they will not be used as part of the recommendation track
>         process.
>   * Peter was the only one who responded and his response was not definitive. 
>       o I am saying this because he questioned the short timeline and said he
>         was working on additional test cases, but never identified the
>         deadline he could work towards.
>   * We are now a week beyond the date set in the “last call for tests” and the
>     WG has been “working out something" even after that date. This has been
>     hindered by the bugs in submitted tests. I would expect that requiring
>     bug-free test submissions does not violate the process. In any case, I
>     think we went over and above by accepting tests with bugs and fixing said
>     bugs.
>   * Obviously, we need to have a “stop date”. With the charter currently set
>     to end on June 1st, I don’t see how this date can be later than today. 
> 
> 
> Overall, I believe that cutting the submission of the tests 5 weeks before the
> end of the period is not “well before the end”. In practice, however, WG
> allowed submission of tests to be used as part of the process for another week
> - cutting it off 4 weeks before the end which is probably the minimal time
> required to go through all the logistics that must happened before the period
> ends. Sandro, of course, as W3C staff member, will be able to tell us if there
> are any process issues.
> 
> Regards,
> 
> Irene Polikoff
> 
> 
> 
>> On May 2, 2017, at 3:53 AM, Holger Knublauch <holger@topquadrant.com
>> <mailto:holger@topquadrant.com>> wrote:
>>
>>
>>
>> On 2/05/2017 1:50, Peter F. Patel-Schneider wrote:
>>> This is a formal objection to the early cutoff of submissions to the working
>>> group's test suite.
>>>
>>> The candidate recommendation period in the W3C process is designed so that
>>> implementers can test out a potential W3C recommendation to see whether
>>> conforming implementations can be created.  Testing to confirm
>>> implementation behaviour is thus an important part of the candidate
>>> recommendation process.  Cutting off the submission of tests well before the
>>> end of the candidate recommendation period hinders wide review of the
>>> candidate recommendation and goes against the purpose of the W3C process.
>>>
>>>
>>> Two tests were submitted that examine the behaviour of SHACL implementations
>>> on property paths that contain information about two different kinds of
>>> SPARQL paths.  These tests came from recent implementation experience.  The
>>> aspect of SHACL that they test is not covered by any of the tests in the
>>> current working group's test suite so without them there is no confirmation
>>> that there are two independent implementations of SHACL property paths.
>>> These tests were not accepted.
>>
>> Since you seem to find this so very important I have now added these two
>> test cases to the suite. I still don't agree they are important, but I don't
>> think this is worth spending any more time on. I have notified the other
>> implementers about the new test(s) and hope they can re-run their tests.
>>
>>>
>>> Two tests were submitted that examine the behaviour of SHACL implementations
>>> of pre-binding.  Pre-binding underlies all of SHACL-SPARQL so it is vitally
>>> important for SHACL-SPARQL both that pre-binding have a suitable definition
>>> and that implementations correctly implement pre-binding.  The working group
>>> approved changes to the definition of pre-binding on 26 April 2017 and the
>>> tests were submitted on 28 April 2017.  These tests replaced previous tests
>>> to examine the behaviour of SHACL implementations of pre-binding that had
>>> become irrelevant because of the change to pre-binding.  These tests were
>>> not accepted.
>>
>> I have just made the use of VALUES in general ill-formed in SHACL-SPARQL,
>> making your submitted test cases outdated. I had never rejected them, see my
>> other email in this thread.
>>
>> So both of the examples that you have used to motivate your formal objection
>> are now handled.
>>
>> Holger
>>
>>
>

Received on Friday, 5 May 2017 19:39:54 UTC