Re: CfC: WebApps testing process; deadline April 20

On 04/21/2011 01:10 AM, Adrian Bateman wrote:
> First, thanks to Art for pulling all this content together. We're looking
> forward to a more structured process for testing as various specifications
> in the WebApps increase in maturity.
>
> I have a couple of small comments related to the issues Aryeh raised.
> Apologies for the lateness of these comments; I spent time sharing this
> process with a number of teams at Microsoft before responding.
>
> 1. Early approval of tests
>
> We think that waiting for Last Call or Candidate Recommendation before
> approving tests loses some of the value of tests in driving ambiguity out
> of the specification. The CSS WG found many issues in CSS 2.1 as a
> consequence of tests and some of these issues we substantive enough that
> the spec went back to Working Draft status. Avoiding this by reviewing test
> cases earlier in the process will help to improve spec quality. I think
> of this in the following way: a bug filed against the spec requesting a
> change represents someone's view that the spec is wrong. On the other hand,
> an approved test with consensus of reviewers in the working group helps to
> identify more stable sections of the spec. It doesn't mean it can't change
> but it does mean the spec has had additional review on the assertions made
> in the test and that's useful.

I agree that late review is not helpful to anyone.

The way I think test review should work is similar to any other form of 
code review:

A user pushes a series of commits to the repository
The user requests review of those commits
Different reviewers comment on the patch
If no problems are found, the tests are considered approved

This encourages early review, makes it obvious what people consider 
stable enough to be reviewed, and allows for sharing of not-yet-ready 
works in progress without having to have multiple public repositories. 
It also eliminates the need for seperate submitted and approved folders 
since approved tests are ones where all the related commits have 
positive review and there are no open bugs.

As far as I can tell the main problem with adopting this workflow is 
that some tooling support is required.

Received on Thursday, 21 April 2011 08:56:23 UTC