Re: CR exit criteria and features at risk for HTML5

On Fri, 17 Aug 2012, L. David Baron wrote:

> On Thursday 2012-08-16 10:32 +0200, James Graham wrote:
>> On 08/16/2012 09:21 AM, Boris Zbarsky wrote:
>>> Well, a good start would be having a simple, advertized way to
>>> contribute tests.  I don't even know how I'd go about contributing a
>>> test for the HTML5 test suite, and my experience with contributing tests
>>> for other W3C working groups has been less than pleasant.  :(
>>>
>>> Of course contributing test suites to browsers that are not Mozilla is
>>> not that easy for me either.  But for Mozilla, it takes me very little
>>> time to add a test to the Mozilla test suite.  If we can get
>>> contributing tests for the official test suite down to that level, I bet
>>> developers will be more likely to contribute.
>>
>> Well the actual "contribute" part is basically "commit to a hg
>> repo". The full instructions are at [1].
>
> Are those instructions sufficient to get a test into the "Approved
> Tests" list? [2]  Or is the "Approved Tests" subset not a relevant
> subset?

My opinion is that the approval system we have has not worked. Where we 
have imported tests at Opera we typically run them just from the submitted 
directory because that contains far more tests, and by actually running 
them we are more likely to find problems than by simply inspecting them 
(or waiting for others to do so). In the long term I think we should 
remove the submitted/ and approved/ directories altogether; the filesystem 
structure is the wrong place to store review metadata. Of course I still 
think that code review is valuable, but I would mcuh rather integrate with 
the VCS and say, for example, that specific commits, or (file, commit) 
tuples have been reviewed.

Also, compared to e.g. the CSS working group, the number of formal 
requirements for tests has been kept intentionally low; although it is 
possible to add similar types of metadata it is not required. Thinking 
about it, an unintentional benefit of dropping the testsuite from the 
Process requirements is that it will become easier to submit tests because 
there will be less pressure to add requirements on test authors desinged 
to make getting implementation reports easy.

Received on Friday, 17 August 2012 20:52:37 UTC