Re: CfC: WebApps testing process; deadline April 20

I agree the need for clear test suite status is implied and should be 
explicit. I've added a new requirement for this to [1]. As to how this 
requirement is addressed, perhaps we should adopt/re-use some existing 
good practice; otherwise perhaps we can use a Status/Readme file in each 
.../tests/ directory.

Besides "WG Approval", you see the need for "maintainer approval". 
Additionally, you think there will be scenarios where multiple 
submissions make it difficult to know which tests have been reviewed 
(e.g. by the maintainer). This could be addressed by the maintainer 
creating a separate directory (e.g. "reviewed") and the maintainer would 
use it to place copies of test cases (e.g. from submissions directories) 
she reviewed (and possibly edited and approved). I don't think the 
process as currently defined would preclude the maintainer from doing 
something like this and if it becomes a common/good practice, we could 
document it. In the case of a single contributor, it's probably not 
something that would be needed (but wouldn't necessarily be harmful).

-Art Barstow

[1] http://www.w3.org/2008/webapps/wiki/Testing#Testing_Goals

On Apr/17/2011 7:06 PM, ext Aryeh Gregor wrote:
> On Wed, Apr 13, 2011 at 10:02 AM, Arthur Barstow<art.barstow@nokia.com>  wrote:
>> I have updated WebApps' testing process documents to reflect comments
>> submitted to the initial draft process [1]. As such, this is a Call for
>> Consensus to agree to the testing process as described in:
>>
>> http://www.w3.org/2008/webapps/wiki/Testing
>> http://www.w3.org/2008/webapps/wiki/Submission
>> http://www.w3.org/2008/webapps/wiki/Approval
>> http://www.w3.org/2008/webapps/wiki/Harness
> Sorry for the lateness of this review -- I was swamped with work and
> didn't find the time to respond earlier.  I still have a few issues
> with the proposed approval procedure.  The way it sounds is that tests
> can either be non-approved, or approved.  Non-approved tests sound
> like (I'm not totally clear on this) they're supposed to live in
> per-contributor submission/ directories, without anyone else
> necessarily having any say over them.  Approved tests, on the other
> hand, only exist at LC or later, and can't be changed without Working
> Group approval.  (Which means what?  I'm not sure.)
>
> The problem I've seen with the submission/ vs. approved/ approach in
> the HTMLWG is that there's no distinction between tests in submission/
> that are useful and correct but just haven't been reviewed by anyone,
> and tests that aren't complete or correct yet and aren't really
> intended for serious use.  We should have a clear test suite that's
> usable in practice well before LC, even if not all the tests are
> reviewed yet.
>
> So what I'd prefer is that the contents of approved/ be under the
> control of the maintainer of the test suite, like the editor controls
> the spec.  If people are submitting tests, the maintainer should be
> allowed to approve them without a CfC, at least while the spec is
> still a Working Draft.  That way we have a single repository from the
> beginning that should include all useful tests, instead of having many
> tests of varying quality scattered throughout submission/.  Once a
> test is in approved/, it should be possible to file bug reports on it,
> discuss it on the mailing list, etc.  It needs to be in a central
> location and the responsibility of the working group, not the
> submitter, so that the working group can ensure the test suite's
> quality from the earliest possible point.
>
> Then a CfC would be whether the WG is okay with the current contents
> of approved/ or whether some of the tests should be un-approved.  A
> CfC shouldn't be necessary to approve tests to begin with, any more
> than it's necessary to make an edit to a spec.

Received on Tuesday, 19 April 2011 16:22:16 UTC