What is the purpose of this group? - was: Re: Who currently executes the tests in the w3c repos?

On Aug 7, 2013, at 9:59 AM, James Graham wrote:

> On 2013-08-07 09:34, Peter Linss wrote:
>> On Aug 7, 2013, at 2:22 AM, Ms2ger wrote:
>> This statement is untrue and misleading. Firstly, our primary focus
>> of testing is getting specs _out_ of CR and into REC, getting to CR
>> does not require tests, but this is not the "only" thing we care about
>> regarding tests.
> 
> I don't know who "our" is supposed to be in this case

The "our" in that context was the CSS WG.

> but it's certainly not my primary goal, and I don't think it should be anyone else's either.

Wow. Ok.

> 
> The whole Process, and indeed the W3C itself, is a means to an end rather than an end in itself. The ultimate goal is to produce useful technologies.

I don't disagree with that sentiment at all, however the W3C has a process, and the CSS WG follows it. That's the way it is, and that's the way it's going to be. The CSS WG is a W3C group, as I believe this one is, and we don't get to arbitrarily decide to ignore the W3C spec process within those groups and continue to call ourselves W3C groups.

I accept that you and others want to change the process and I have no issue with that, but please take those efforts to the appropriate forum, like the AC. If the process changes and REC is no longer deemed to have any value, or tests are not required to achieve REC status, I'm fine with that. But until that happens, the CSS WG is going to follow the existing W3C process and drive our specs to REC.

At this point I need to re-ask the question and insist on a clear, definitive, answer: Is this effort, by which I mean the W3C central test repository and the testing infrastructure built around it, going to support the CSS WG's needs to get specs to REC or not?

If the answer to that is no, fine, but then I'm not going to waste any more of my time trying to merge the CSS WG test repo with the central repo as that would be defeating the CSS WG's goals. We've run for years with our own repo and our own tools and we can continue to do so going forward. We can still share parts of our technology were practical, but our primary focus will be on meeting our own needs.

If the answer is yes, fine too, but then let's stop this debate and focus on what it's going to take to meet those needs.



> One of the ways that we make the technologies useful is by allowing multiple, interoperable, implementations, which means that we need precise specification, and careful, thorough testing. The Process is supposed to be there to help achieve these things, but it is at best a weak proxy for the underlying requirements. Optimising for the proxy rather than the actual requirements is actively harmful and we have to be careful not to do it. For example, removing tests that don't pass from the testsuite is helpful in getting to Rec. but detrimental to interoperability. But this is something I have seen people do.

I can't speak to other WGs but in the CSS WG we have removed _features_ from a spec that weren't ready so that the rest of the spec could advance on the REC track (and generally move those features to a future level of the spec). When that happens it is appropriate for the tests of that feature to move out of that spec's test suite and get put elsewhere. Personally, I see the value in having a stable REC document with a matching test suite. I see the value in other specification approaches too, but as I said above, this isn't the place to debate the merits of one approach vs the other.

> 
>> Secondly, this implies that we're gaming the system and don't care
>> about the quality of the tests we use, which is completely false. We
>> used to have a policy of not including tests in our suites until they
>> were reviewed. We found that our review backlog was too long (by
>> thousands of tests) and changed our review policy to allow unreviewed
>> tests into the suites and allow the reviews to be triggered by looking
>> for unexpected results. Yes, there are most likely a small percentage
>> of tests that are incorrect, but the review backlog is still being
>> worked even after the relevant spec has exited CR and the test suite
>> shifted into the conformance stage.
> 
> FWIW I see the arguments in favour of such post-hoc review as being more efficient. However the long term goal is to get to a place where people are submitting tests directly to the W3C that they wish to include in their own implementation testsuites. Code going into browsers would be reviewed in any case, so it is no extra effort to do this review in public as part of the W3C submission process compared to doing it in their own review systems. Of course contributions from external contributers will need extra review, but I don't think that's a big time sink compared to the value.

I have no issue with attempting to go back to the pre-inclusion review process for the CSS WG. I'm just pointing out that in the past it was necessary to change that, and it may prove to be the case again, for both the CSS WG and others. When tests are delivered in small increments, the pre-review can be very effective, when tests are delivered by the thousands, the review bottleneck can become overwhelming. The test infrastructure shouldn't be structured in such a way that changing the review process itself becomes an insurmountable task. Taking the time now to at least have a plan in place for that contingency would be time well spent.

Received on Wednesday, 7 August 2013 19:38:13 UTC