Re: Who currently executes the tests in the w3c repos?

On 2013-08-07 09:34, Peter Linss wrote:
> On Aug 7, 2013, at 2:22 AM, Ms2ger wrote:
> 
> This statement is untrue and misleading. Firstly, our primary focus
> of testing is getting specs _out_ of CR and into REC, getting to CR
> does not require tests, but this is not the "only" thing we care about
> regarding tests.

I don't know who "our" is supposed to be in this case but it's 
certainly not my primary goal, and I don't think it should be anyone 
else's either.

The whole Process, and indeed the W3C itself, is a means to an end 
rather than an end in itself. The ultimate goal is to produce useful 
technologies. One of the ways that we make the technologies useful is by 
allowing multiple, interoperable, implementations, which means that we 
need precise specification, and careful, thorough testing. The Process 
is supposed to be there to help achieve these things, but it is at best 
a weak proxy for the underlying requirements. Optimising for the proxy 
rather than the actual requirements is actively harmful and we have to 
be careful not to do it. For example, removing tests that don't pass 
from the testsuite is helpful in getting to Rec. but detrimental to 
interoperability. But this is something I have seen people do.

> Secondly, this implies that we're gaming the system and don't care
> about the quality of the tests we use, which is completely false. We
> used to have a policy of not including tests in our suites until they
> were reviewed. We found that our review backlog was too long (by
> thousands of tests) and changed our review policy to allow unreviewed
> tests into the suites and allow the reviews to be triggered by looking
> for unexpected results. Yes, there are most likely a small percentage
> of tests that are incorrect, but the review backlog is still being
> worked even after the relevant spec has exited CR and the test suite
> shifted into the conformance stage.

FWIW I see the arguments in favour of such post-hoc review as being 
more efficient. However the long term goal is to get to a place where 
people are submitting tests directly to the W3C that they wish to 
include in their own implementation testsuites. Code going into browsers 
would be reviewed in any case, so it is no extra effort to do this 
review in public as part of the W3C submission process compared to doing 
it in their own review systems. Of course contributions from external 
contributers will need extra review, but I don't think that's a big time 
sink compared to the value.

Received on Wednesday, 7 August 2013 17:01:23 UTC