Re: DOM Test Suite task force

Curt Arnold wrote:

> 
> build and schema/DTD generation are probably worthy to be components.
> 

That sounds like a good idea.

http://www.w3.org/Bugs/Public/show_bug.cgi?id=189

Please detail what areas each component would cover and either volunteer 
or nominate an owner for these proposed components.

> There were good intentions to do transforms and frameworks for other 
> "unofficial" bindings (Python, Xerces-C, libxml, System.Xml, et al).  It 
> would be good to clarify whether they can be done within the context of 
> the W3C effort and if so to define components and a process for 
> developing or accepting test suite generation for other bindings.
> 

I have indeed already had inquiries regarding this. I think that we 
should work to enable others to reuse the test suite to test other 
implementations. However, I think that this topic needs more guidance 
from the WG on how to proceed. Our initial goal is to complete and 
release the DOM TS and I would not want this worthy topic to detract 
from that goal.

I will start a discussion here to discuss what we can and should do with 
respect to "unofficial" bindings.

> It might be better to organize the components by sub-spec (Core, HTML, 
> Events, LS, XPath) than by level.  The Level 1 owner isn't going to have 
> much to do.
>

I agree that it would be very nice to be able to specify the 
sub-specification however I also wanted to make the level explicit to 
help identify the version of the test suite affected by the bug/issue.

Ideally we would have components for each level/spec however the benefit 
of being able to assign bugs/issues to that level of detail seemed to be 
outweighed by the sheer number of required components and owners. Of 
course we can have a single individual who owns multiple components.

I am open to expanding the components to include the sub-specification 
however I want to keep the level as well.

The fact that the DOM Level 1 owner (and hopefully the DOM Level 2 
owner) will hopefully have a short time of activity is the goal of our 
project.

The question is would

DOM Level 1
DOM Level 2 Core
DOM Level 2 HTML
....

be acceptable or would this create too many components?

> 
> 
> 
> That might be appropriate for the later phases of a test suite's life, 
> but it would seem to severely hinder collaboration in the development of 
> new test suites.  I've been troubled that there has been very little 
> visibility and collaboration in the development of the initial LS and 
> XPath test suites and such a lock-down would reinforce the tendency for 
> new test suites to be developed in hiding.
>

I agree with your basic point. I think that we can consider DOM Level 1 
and DOM Level 2 to be at the stage where more controls are appropriate 
but that DOM Level 3 is not.

> There needs to be some middle ground between tests on one developer's 
> machine and code that has implied sanction of the WG.  If our releases 
> were more frequent, then we could tolerate the CVS containing a few 
> controversial tests awaiting review by the WG.  To have more frequent 
> releases, we would need some incremental status for releases.  The 
> progression could be from a nightly build, to working-draft release, to 
> a last-call release that the task force and WG approved to a definitive 
> release.  Has there been anything in the QA WG on test suite lifecycles 
> that might be appropriate?

I do not know if the QA WG has anything appropriate however I do 
envision creating a system where we do have public nightly builds 
available for testing and review and agree with your general process 
outline.

These are issues that I think we can handle in the overall project 
framework once we have the basic initial process defined and have owners 
for the basic components.

> 
> Except for code freeze periods, the review before commit seems terribly 
> pessimistic.  If a committer is considering something controversial or 
> that touches a lot of files, good manners would suggest that it be 
> brought up on the mailing list for discussion before being committed. In 
> the, hopefully rare, case that a committer did not think a particular 
> change was controversial, but it was, it would not be difficult to undo 
> the change until there was consensus.  Maybe I'm insensitive since I 
> would be the most likely offender, but I do not believe that we have had 
>  abuses of committer priviledges to warrent such a strong corrective.
> 

For a sub-specification that is in it's earliest stages such as DOM 
Level 3, I agree that mandatory review is not absolutely necessary.

However for other sub-specifications which are in their final stages I 
do believe that mandatory reviews are necessary.

I also think that other aspects of the project need to be under tighter 
control and can benefit from a mandatory review process. For example, 
the ECMAScript binding has been an area of disagreement and we are not 
in a position of saying we have decided how to proceed or complete it. 
The process to date has been for myself and others to raise points, 
provide demonstrations of ideas, and solicit feedback. However you make 
the final decisions as to the exact changes to the ECMAScript binding 
and commit them to CVS without any discussion or approval of what those 
specific changes will be.

I believe that each Bugzilla component can have separate review 
requirements. At the earliest stages, no review would be needed. As the 
Component nears completion, mandatory review can be required.

> The most effective review of a proposed change would be a nightly (or 
> more frequent build) that would validate all tests against the 
> appropriate schema, build the bindings, and possibly run the tests 
> against benchmark implementations (which would probably be limited to 
> Java bindings at this time).  To effectively review before commit, the 
> reviewer would be forced to apply the patch to his local copy and run 
> through such a build process semi-automatically.  To review after 
> commit, the reviewer could rely on the automated build to find any 
> structural problems with the change and would only need to verify that 
> the change is consistent with the spec.  I would not suspect that it 
> would be difficult to get the DOM TS added to Gump's 
> (http://jakarta.apache.org/gump) roster.

I do envision a system of nightly builds, candidate builds, etc however 
I do not think they constitute a replacement for the need to review 
changes to those parts of the TS which are nearing completion.

> 
> 
> Component Owners seem a generally good idea.  However, if, for example, 
> a change was committed in the DOM L3 TS that broke the build or was not 
> schema valid, it would be good for a committer to fix the problem 
> without requiring an assignment of the defect by the component owner.

I think that if someone breaks the build and notices it before anyone 
else, the person responsible for the change should consider it their 
immediate responsibility to backout the commit that caused the breakage 
and to reopen the bug which tracked that change.

If someone else notices that the build is broken due to another person's 
commit, then the bug which tracks the change should be reopened and the 
person responsible for the change notified and given the chance to 
correct the problem.  If the person responsible for the change can not 
respond quickly enough, those with the appropriate permissions should 
backout the change without waiting for the original person.

The basic underlying point here is that no changes should be made 
without tracking that change in a bug whether or not the component in 
question requires review or not.

A new bug is not required in order to back out a commit which breaks the 
build. Reopening the already existing bug is sufficient.

> 
> Breaking down QA by components might be overkill, maybe an QA role that 
> can mark any bug as closed with an possible informal division of 
> responsibilities.
> 
> There should be no criteria to be a Reviewer (in the monitor change, not 
> the veto change sense) or a general contributor.  Anyone who wants to 
> subscribe to www-dom-ts@w3.org or access the CVS should be encouraged to 
> do so.
> 

I agree that anyone who wishes to participate is welcome and should be 
encouraged. However the role of "reviewer" who accepts/rejects/asks for 
modifications to a change before it is committed implies knowledge and 
responsibilities which are not those of a general contributor.

In our context "review" and "reviewer" have specific meaning related to 
controlling changes committed to CVS to make sure that the change is 
appropriate.

> Those with commit rights to the CVS (aka Developers) should be a 
> separate class.
> 
> I'd see the roles break down like:
> 
> Community/Public: Anyone interested in the DOM TS work.  Can subscribe 
> and post to www-dom-ts, access the CVS, subscribe to CVS change 
> notifications, access Bugzilla.
> 
> Developers: Public rights + CVS commit rights.  May research bugs and 
> commit changes to CVS corresponding to bugs.  Should not make 
> potentially controversial changes to CVS without raising the issue on 
> www-dom-ts.   When there is a controversy (either before or after a CVS 
> change), any committer can block (or rollback) the change until WG or WG 
> liason resolution, accept the change pending WG or WG liason resolution 
> or accept the change without escalation to the WG liason.  Votes on task 
> force interim releases.
> 

Any change should first be tracked by a bug. If necessary, the bug can 
be announced on the www-dom-ts to solicit a discussion but if there is 
controversy then the comments in the bug should reflect that and the 
commit not made to CVS until the issue has been resolved either by the 
Component owner, the DOM TS chair or the DOM WG.

Under no circumstances would I feel that it is ok to check in a change 
to CVS if there has been disagreement with either the Component owner, 
the DOM TS chair or the DOM WG.

> QA: Monitors Bugzilla, can mark issues as closed.
> 
> WG liason: Can block or rollback changes to the CVS.  Coordinates issues 
>  resolution with the WG.  Votes on task force interim releases.
> 
> WG: approves formal releases of test suite.
> 
> The most crucial role that we have been missing coverage has been 
> implementation representatives.  It would be great to have someone who 
> is the go to person for issues specific to Xerces-J, Safari, Konqueror, 
> Palm Web Browser, etc.
> 

I have preliminary contacts with Safari and Opera and hope to make the 
participation of implementation representatives as broad as possible.