Re: formal objection to early cutoff of test submissions

Peter,

The decision to accept tests submitted after the deadline, should the WG vote to include them, was minuted in the https://www.w3.org/2017/05/03-shapes-minutes.html <https://www.w3.org/2017/05/03-shapes-minutes.html>.

Do you see this as a sufficient announcement and are, therefore, withdrawing your formal objection? If not, let us know what you would see as a sufficient announcement.

Regards,

Irene

> On May 5, 2017, at 3:39 PM, Peter F. Patel-Schneider <pfpschneider@gmail.com> wrote:
> 
> 
> 
> It appears the the working group has changed its policy and will now
> continue to accept test suite submissions subject to a vote of the working
> group.  If the policy on test suite submissions has changed and this change
> in policy is announced, this formal objection is moot.
> 
> Peter F. Patel-Schneider
> Nuance Communications
> 
> 
> On 05/02/2017 09:58 AM, Irene Polikoff wrote:
>> To me, this sounds like a procedural objection:
>> 
>>>> Cutting off the submission of tests well before the
>>>> end of the candidate recommendation period hinders wide review of the
>>>> candidate recommendation and goes against the purpose of the W3C process.
>> 
>> I think the key phrase is "well before the end of the candidate recommendation
>> period”. 
>> 
>> I believe the process has been followed:
>> 
>>  * Obviously, submission of tests to be used as part of the recommendation
>>    track process needs to be concluded before the end of the end of the
>>    candidate recommendation period.
>>      o This is because implementers need time to run all tests and make fixes
>>        if necessary.
>>      o Then the WG needs to review the results, draft the transition request
>>        and decide to submit it.
>>      o Then request to transition has to be sent and transition meeting has
>>        to happen. 
>>      o All these activities have to occur before the candidate recommendation
>>        period ends. Conservatively speaking, they take at least a month.
>>  * Further, the WG issued a “last call” for tests asking any interested
>>    parties to let us know if they need extra time and, if so, how much. We
>>    promised to “work something out” if such parties tell us about their plans
>>    for submitting
>>    tests https://lists.w3.org/Archives/Public/public-rdf-shapes/2017Apr/0034.html
>>      o The last call also said that additional tests can be submitted at any
>>        time - as something for the implementers to test with, but, after the
>>        deadline, they will not be used as part of the recommendation track
>>        process.
>>  * Peter was the only one who responded and his response was not definitive. 
>>      o I am saying this because he questioned the short timeline and said he
>>        was working on additional test cases, but never identified the
>>        deadline he could work towards.
>>  * We are now a week beyond the date set in the “last call for tests” and the
>>    WG has been “working out something" even after that date. This has been
>>    hindered by the bugs in submitted tests. I would expect that requiring
>>    bug-free test submissions does not violate the process. In any case, I
>>    think we went over and above by accepting tests with bugs and fixing said
>>    bugs.
>>  * Obviously, we need to have a “stop date”. With the charter currently set
>>    to end on June 1st, I don’t see how this date can be later than today. 
>> 
>> 
>> Overall, I believe that cutting the submission of the tests 5 weeks before the
>> end of the period is not “well before the end”. In practice, however, WG
>> allowed submission of tests to be used as part of the process for another week
>> - cutting it off 4 weeks before the end which is probably the minimal time
>> required to go through all the logistics that must happened before the period
>> ends. Sandro, of course, as W3C staff member, will be able to tell us if there
>> are any process issues.
>> 
>> Regards,
>> 
>> Irene Polikoff
>> 
>> 
>> 
>>> On May 2, 2017, at 3:53 AM, Holger Knublauch <holger@topquadrant.com
>>> <mailto:holger@topquadrant.com>> wrote:
>>> 
>>> 
>>> 
>>> On 2/05/2017 1:50, Peter F. Patel-Schneider wrote:
>>>> This is a formal objection to the early cutoff of submissions to the working
>>>> group's test suite.
>>>> 
>>>> The candidate recommendation period in the W3C process is designed so that
>>>> implementers can test out a potential W3C recommendation to see whether
>>>> conforming implementations can be created.  Testing to confirm
>>>> implementation behaviour is thus an important part of the candidate
>>>> recommendation process.  Cutting off the submission of tests well before the
>>>> end of the candidate recommendation period hinders wide review of the
>>>> candidate recommendation and goes against the purpose of the W3C process.
>>>> 
>>>> 
>>>> Two tests were submitted that examine the behaviour of SHACL implementations
>>>> on property paths that contain information about two different kinds of
>>>> SPARQL paths.  These tests came from recent implementation experience.  The
>>>> aspect of SHACL that they test is not covered by any of the tests in the
>>>> current working group's test suite so without them there is no confirmation
>>>> that there are two independent implementations of SHACL property paths.
>>>> These tests were not accepted.
>>> 
>>> Since you seem to find this so very important I have now added these two
>>> test cases to the suite. I still don't agree they are important, but I don't
>>> think this is worth spending any more time on. I have notified the other
>>> implementers about the new test(s) and hope they can re-run their tests.
>>> 
>>>> 
>>>> Two tests were submitted that examine the behaviour of SHACL implementations
>>>> of pre-binding.  Pre-binding underlies all of SHACL-SPARQL so it is vitally
>>>> important for SHACL-SPARQL both that pre-binding have a suitable definition
>>>> and that implementations correctly implement pre-binding.  The working group
>>>> approved changes to the definition of pre-binding on 26 April 2017 and the
>>>> tests were submitted on 28 April 2017.  These tests replaced previous tests
>>>> to examine the behaviour of SHACL implementations of pre-binding that had
>>>> become irrelevant because of the change to pre-binding.  These tests were
>>>> not accepted.
>>> 
>>> I have just made the use of VALUES in general ill-formed in SHACL-SPARQL,
>>> making your submitted test cases outdated. I had never rejected them, see my
>>> other email in this thread.
>>> 
>>> So both of the examples that you have used to motivate your formal objection
>>> are now handled.
>>> 
>>> Holger
>>> 
>>> 
>>