For comments, an outline of a CSS test suite review system

Hi, there,

CSS 2.1 test suite development is becoming a high priority work item, especially with CSS WG's focus moving 2.1 to PR. Brainstorming with a number of people actively involved here, an outline of an so-called Ideal (or Dream, in fantasai's lingual) Test Review System starts to emerge. The three part descroption below is a sketch of it. We are looking for comments and/or suggestions from the CSS testsuite community.

Some key objectives identified are Open, Transparent, Enabling and with High Quality standard: Open in term of public involvement, Transparent regarding how the process work, Enabling for low overhead for submitter/reviewers/Admin, and High Quality, so avoiding g*-in-g*-out.

Part 1. There are four roles in the review system.
1.      Anonymous user
i.e. everyone who are interested in contributing to CSS test suite review; they can have access to the submitted test cases and related comments, and provide their own comments on the test cases.

2.      Registered user
The registration should be open to anyone interested. Those who registered, can submit their test case(s), make comments; they can change the state of a test case with comments hopefully. (see test case state list below for details)

3.      Registered user with modification privilege
These are ones who are assigned to an ACL that allow them to make modification to test cases (to incorporate certain review comments). A test case submitter is by default, such user who can modify his/her test cases. Others could be assigned such privilege to make modification, especially when submitters are not available to make the timely changes.

4.      Admin user
Admin are the ones CSS WG has appointed to have the permission to ultimately check in the test cases.  They are also called Approvers.

Part 2. A test case has the following six valid states in the review system
1.      Submitted
2.      Rejected: need-fix
3.      Rejected: duplication
4.      Rejected: out-of-scope
5.      Accepted
6.      Checked-in

Any user can change test case state, moving it around among any of #1 to #5 states, at any time, though only Admin can move a test case to "Checked-in" state.  A public log (per test case) will be availableto to show the test case's every state changes and every comments provided, in chronological order.

Any modification to a test case will also produce a timestamp, the user id and if available, the comment, to the public log of the test case, in a similar way as a comment submitted.

Part 3. Lastly, regarding infrastructure, we would like to see the following.
The infrastructure allows easy test case submission; They will allow submission of individual test case, zip file which contains multiple test case files (or even directories), or even an url to an individual test case hosted outside W3C. Once submitted, the test cases will then (automatically) be individually presented to public for comments or changes.

The infrastructure will also group test cases by their state in the review system, and enable straightforward access to each group of test cases, for casual reviewers and serious contributors alike.



If you have any comments, suggestion or ideas, please feel free to share with the group. I hope that you will find most of what we describe here are either current practice or based on common sense.

We call it an Ideal or Dream system, because we are trying to be realistic:) If we are so lucky, there might be a system in W3C or outside, that can be quickly retrofit to implement what will be finally decided by the community; otherwise (and in the most likely scenario), we have to get by with some existing system with work-around and duct-tapes (hoping that over-time, infrastructure will catch up). Also, when Arron and I initially discussed, we had a one-wiki-page-per-test-case scenario in mind. Though in this description the wiki-part is not emphasized, it is a likely approach.

Br,

Ming

Ming Gao

Imaging and Printing Group
Hewlett Packard Company
16399 W Bernardo Dr.
San Diego, CA 92127-1899
United States

Received on Saturday, 19 April 2008 20:34:03 UTC