Re: DOM Test Suite task force

Robert Clary wrote:
> 
> Contents
> 
>   DOM Test Suite task force
>   DOM Test Suite Process
>     Bugzilla
>     Change Control
>   Call for Volunteers/Nominations
> 
> 
> DOM Test Suite task force
> ==================================================
> 
> I have been invited to chair the DOM Test Suite task force
> with the objective of managing the work required to complete
> and release the DOM Test Suites for DOM Levels 1, 2 and 3.
> 
> I would like to outline some of the basic changes in how
> the project will be managed and ask for nominations and
> volunteers to help complete the DOM TS.
> 
> Please feel free to make comments or suggestions regarding
> these changes.
> 
> Changes to the DOM Test Suite Process
> ==================================================
> 
> The existing procedures for the DOM TS are outlined in
> <http://www.w3.org/DOM/DOMTS-Process>.
> 
> In order to for the DOM TS to be completed, we need to know
> what work needs to be completed, who is responsible for
> completing the work and have some degree of control over
> what changes are made to the project.
> 
> In order to achieve this, we will manage all work through
> Bugzilla.
> 
> <www-dom-ts@w3.org> will continue to be used for general
> discussion.
> 
> Bugzilla
> --------------------------------------------------
> 
> The public W3 instance of Bugzilla <http://www.w3.org/Bugs/Public/>
> will be used to identify and track each task required for the
> completion of the DOM TS.
> 
> A new Bugzilla product "DOM TS" and the following Components
> have been created:
> 
> Component     Description
> ---------     -----------
> Process       This component would be used to manage changes
>               in the process used to manage DOM TS project.
> 
> Documentation This component would be used to manage the
>               <http://www.w3.org/DOM/Test> site as well as
>               any other documentation related to the DOM Test
>               suites.
> 
>               Any changes to the public web site will be made
>               by Philippe Le Hegaret after approval of the
>               full document by the DOM WG.
> 
> Java Binding  This component would manage the maintenance of
>               the XSLT and build processes used to create the
>               binding of the tests to the Java Test harness.
> 
> ECMAScript    This component would manage the maintenance of
> Binding       the XSLT and build processes used to create the
>               binding of the tests to the ECMAScript Test harness.
> 
> DOM Level 1   This component would be for all issues related
>               to the DOM Level 1 test suite.
> 
> DOM Level 2   This component would be for all issues related
>               to the DOM Level 2 test suite.
> 
> DOM Level 3   This component would be for all issues related
>               to the DOM Level 3 test suite.

build and schema/DTD generation are probably worthy to be components.

There were good intentions to do transforms and frameworks for other 
"unofficial" bindings (Python, Xerces-C, libxml, System.Xml, et al).  It 
would be good to clarify whether they can be done within the context of 
the W3C effort and if so to define components and a process for 
developing or accepting test suite generation for other bindings.

It might be better to organize the components by sub-spec (Core, HTML, 
Events, LS, XPath) than by level.  The Level 1 owner isn't going to have 
much to do.


> The Component Owner is responsible for either performing the
> work required to resolve the bug or for assigning the bug to
> an appropriate member of the DOM TS community and scheduling
> it to be resolved.
> 
> Ownership of each DOM TS Component has been temporarily assigned
> to myself until suitable owners can be identified.
> 
> Change Control
> --------------------------------------------------
> 
> If we are to complete the DOM TS, each change to the TS must
> be managed. Therefore, before any change is committed to the
> DOM TS:
> 
>   * The reason for the change must be documented as a bug in
>     Bugzilla
> 
>   * For changes to existing files, the change must be attached
>     to the bug as a patch. For new files, the files must be
>     attached to the bug.
> 
>   * Each change must be reviewed by a person who has been
>     given reviewer status. If the reviewer suggests modifications
>     or otherwise does not approve the change to the DOM TS,
>     then the change must not be checked into CVS.

That might be appropriate for the later phases of a test suite's life, 
but it would seem to severely hinder collaboration in the development of 
new test suites.  I've been troubled that there has been very little 
visibility and collaboration in the development of the initial LS and 
XPath test suites and such a lock-down would reinforce the tendency for 
new test suites to be developed in hiding.

There needs to be some middle ground between tests on one developer's 
machine and code that has implied sanction of the WG.  If our releases 
were more frequent, then we could tolerate the CVS containing a few 
controversial tests awaiting review by the WG.  To have more frequent 
releases, we would need some incremental status for releases.  The 
progression could be from a nightly build, to working-draft release, to 
a last-call release that the task force and WG approved to a definitive 
release.  Has there been anything in the QA WG on test suite lifecycles 
that might be appropriate?

Except for code freeze periods, the review before commit seems terribly 
pessimistic.  If a committer is considering something controversial or 
that touches a lot of files, good manners would suggest that it be 
brought up on the mailing list for discussion before being committed. 
In the, hopefully rare, case that a committer did not think a particular 
change was controversial, but it was, it would not be difficult to undo 
the change until there was consensus.  Maybe I'm insensitive since I 
would be the most likely offender, but I do not believe that we have had 
  abuses of committer priviledges to warrent such a strong corrective.

The most effective review of a proposed change would be a nightly (or 
more frequent build) that would validate all tests against the 
appropriate schema, build the bindings, and possibly run the tests 
against benchmark implementations (which would probably be limited to 
Java bindings at this time).  To effectively review before commit, the 
reviewer would be forced to apply the patch to his local copy and run 
through such a build process semi-automatically.  To review after 
commit, the reviewer could rely on the automated build to find any 
structural problems with the change and would only need to verify that 
the change is consistent with the spec.  I would not suspect that it 
would be difficult to get the DOM TS added to Gump's 
(http://jakarta.apache.org/gump) roster.


>   * Changes committed to CVS must include a comment containing
>     the bug number, the reviewer and a short description of
>     the change.
> 
> A hierarchy of component owner, reviewer, DOM TS chair, DOM WG
> will be responsible for making decisions regarding changes to
> the DOM TS.
> 
> If at any point, a decision is questioned, the decision will
> be bubbled up to the next highest level of authority.
> 
> The final authority for any decision regarding the DOM TS is
> the DOM WG.
> 
> Repeated violations of the CVS check in rules would be grounds
> for revocation of CVS commit privileges.
> 
> Call for Volunteers/Nominations
> ==================================================
> 
> We need people to help complete the DOM TS as:
> 
>   Component Owners
> 
>     Component Owners are responsible for managing the work
>     involved in each component.
> 
>   Component QA
> 
>     Component QA are responsible for verifying that RESOLVED
>     bugs/issues in Bugzilla are in fact resolved
>     as described in the bug.
> 
>   Reviewers
> 
>     Reviewers are responsible for ensuring that any changes made
>     to the DOM TS are appropriate.
> 
>   General Contributors
> 
>     General Contributors help by filing new bugs, helping to
>     resolve existing bugs or by contributing documentation.

Component Owners seem a generally good idea.  However, if, for example, 
a change was committed in the DOM L3 TS that broke the build or was not 
schema valid, it would be good for a committer to fix the problem 
without requiring an assignment of the defect by the component owner.

Breaking down QA by components might be overkill, maybe an QA role that 
can mark any bug as closed with an possible informal division of 
responsibilities.

There should be no criteria to be a Reviewer (in the monitor change, not 
the veto change sense) or a general contributor.  Anyone who wants to 
subscribe to www-dom-ts@w3.org or access the CVS should be encouraged to 
do so.

Those with commit rights to the CVS (aka Developers) should be a 
separate class.

I'd see the roles break down like:

Community/Public: Anyone interested in the DOM TS work.  Can subscribe 
and post to www-dom-ts, access the CVS, subscribe to CVS change 
notifications, access Bugzilla.

Developers: Public rights + CVS commit rights.  May research bugs and 
commit changes to CVS corresponding to bugs.  Should not make 
potentially controversial changes to CVS without raising the issue on 
www-dom-ts.   When there is a controversy (either before or after a CVS 
change), any committer can block (or rollback) the change until WG or WG 
liason resolution, accept the change pending WG or WG liason resolution 
or accept the change without escalation to the WG liason.  Votes on task 
force interim releases.

QA: Monitors Bugzilla, can mark issues as closed.

WG liason: Can block or rollback changes to the CVS.  Coordinates issues 
  resolution with the WG.  Votes on task force interim releases.

WG: approves formal releases of test suite.

The most crucial role that we have been missing coverage has been 
implementation representatives.  It would be great to have someone who 
is the go to person for issues specific to Xerces-J, Safari, Konqueror, 
Palm Web Browser, etc.

> 
> If you would like to be involved in one of these areas or know of
> someone you think would be a good candidate, please contact me.
> 
> Bob Clary

Received on Wednesday, 16 April 2003 02:07:16 UTC