- From: James Graham <james@hoppipolla.co.uk>
- Date: Wed, 07 Aug 2013 09:59:19 -0700
- To: <public-test-infra@w3.org>
On 2013-08-07 09:34, Peter Linss wrote: > On Aug 7, 2013, at 2:22 AM, Ms2ger wrote: > > This statement is untrue and misleading. Firstly, our primary focus > of testing is getting specs _out_ of CR and into REC, getting to CR > does not require tests, but this is not the "only" thing we care about > regarding tests. I don't know who "our" is supposed to be in this case but it's certainly not my primary goal, and I don't think it should be anyone else's either. The whole Process, and indeed the W3C itself, is a means to an end rather than an end in itself. The ultimate goal is to produce useful technologies. One of the ways that we make the technologies useful is by allowing multiple, interoperable, implementations, which means that we need precise specification, and careful, thorough testing. The Process is supposed to be there to help achieve these things, but it is at best a weak proxy for the underlying requirements. Optimising for the proxy rather than the actual requirements is actively harmful and we have to be careful not to do it. For example, removing tests that don't pass from the testsuite is helpful in getting to Rec. but detrimental to interoperability. But this is something I have seen people do. > Secondly, this implies that we're gaming the system and don't care > about the quality of the tests we use, which is completely false. We > used to have a policy of not including tests in our suites until they > were reviewed. We found that our review backlog was too long (by > thousands of tests) and changed our review policy to allow unreviewed > tests into the suites and allow the reviews to be triggered by looking > for unexpected results. Yes, there are most likely a small percentage > of tests that are incorrect, but the review backlog is still being > worked even after the relevant spec has exited CR and the test suite > shifted into the conformance stage. FWIW I see the arguments in favour of such post-hoc review as being more efficient. However the long term goal is to get to a place where people are submitting tests directly to the W3C that they wish to include in their own implementation testsuites. Code going into browsers would be reviewed in any case, so it is no extra effort to do this review in public as part of the W3C submission process compared to doing it in their own review systems. Of course contributions from external contributers will need extra review, but I don't think that's a big time sink compared to the value.
Received on Wednesday, 7 August 2013 17:01:23 UTC