Re: formal objection to early cutoff of test submissions

To me, this sounds like a procedural objection:

>> Cutting off the submission of tests well before the
>> end of the candidate recommendation period hinders wide review of the
>> candidate recommendation and goes against the purpose of the W3C process.

I think the key phrase is "well before the end of the candidate recommendation period”. 

I believe the process has been followed:

Obviously, submission of tests to be used as part of the recommendation track process needs to be concluded before the end of the end of the candidate recommendation period.
This is because implementers need time to run all tests and make fixes if necessary.
Then the WG needs to review the results, draft the transition request and decide to submit it.
Then request to transition has to be sent and transition meeting has to happen. 
All these activities have to occur before the candidate recommendation period ends. Conservatively speaking, they take at least a month.
Further, the WG issued a “last call” for tests asking any interested parties to let us know if they need extra time and, if so, how much. We promised to “work something out” if such parties tell us about their plans for submitting tests https://lists.w3.org/Archives/Public/public-rdf-shapes/2017Apr/0034.html <https://lists.w3.org/Archives/Public/public-rdf-shapes/2017Apr/0034.html>
The last call also said that additional tests can be submitted at any time - as something for the implementers to test with, but, after the deadline, they will not be used as part of the recommendation track process.
Peter was the only one who responded and his response was not definitive. 
I am saying this because he questioned the short timeline and said he was working on additional test cases, but never identified the deadline he could work towards.
We are now a week beyond the date set in the “last call for tests” and the WG has been “working out something" even after that date. This has been hindered by the bugs in submitted tests. I would expect that requiring bug-free test submissions does not violate the process. In any case, I think we went over and above by accepting tests with bugs and fixing said bugs.
Obviously, we need to have a “stop date”. With the charter currently set to end on June 1st, I don’t see how this date can be later than today. 

Overall, I believe that cutting the submission of the tests 5 weeks before the end of the period is not “well before the end”. In practice, however, WG allowed submission of tests to be used as part of the process for another week - cutting it off 4 weeks before the end which is probably the minimal time required to go through all the logistics that must happened before the period ends. Sandro, of course, as W3C staff member, will be able to tell us if there are any process issues.

Regards,

Irene Polikoff



> On May 2, 2017, at 3:53 AM, Holger Knublauch <holger@topquadrant.com> wrote:
> 
> 
> 
> On 2/05/2017 1:50, Peter F. Patel-Schneider wrote:
>> This is a formal objection to the early cutoff of submissions to the working
>> group's test suite.
>> 
>> The candidate recommendation period in the W3C process is designed so that
>> implementers can test out a potential W3C recommendation to see whether
>> conforming implementations can be created.  Testing to confirm
>> implementation behaviour is thus an important part of the candidate
>> recommendation process.  Cutting off the submission of tests well before the
>> end of the candidate recommendation period hinders wide review of the
>> candidate recommendation and goes against the purpose of the W3C process.
>> 
>> 
>> Two tests were submitted that examine the behaviour of SHACL implementations
>> on property paths that contain information about two different kinds of
>> SPARQL paths.  These tests came from recent implementation experience.  The
>> aspect of SHACL that they test is not covered by any of the tests in the
>> current working group's test suite so without them there is no confirmation
>> that there are two independent implementations of SHACL property paths.
>> These tests were not accepted.
> 
> Since you seem to find this so very important I have now added these two test cases to the suite. I still don't agree they are important, but I don't think this is worth spending any more time on. I have notified the other implementers about the new test(s) and hope they can re-run their tests.
> 
>> 
>> Two tests were submitted that examine the behaviour of SHACL implementations
>> of pre-binding.  Pre-binding underlies all of SHACL-SPARQL so it is vitally
>> important for SHACL-SPARQL both that pre-binding have a suitable definition
>> and that implementations correctly implement pre-binding.  The working group
>> approved changes to the definition of pre-binding on 26 April 2017 and the
>> tests were submitted on 28 April 2017.  These tests replaced previous tests
>> to examine the behaviour of SHACL implementations of pre-binding that had
>> become irrelevant because of the change to pre-binding.  These tests were
>> not accepted.
> 
> I have just made the use of VALUES in general ill-formed in SHACL-SPARQL, making your submitted test cases outdated. I had never rejected them, see my other email in this thread.
> 
> So both of the examples that you have used to motivate your formal objection are now handled.
> 
> Holger
> 
>