Thoughts on restricting non-bugzilla-driven spec changes during Last Call

Hello HTML Working Group,

One topic of discussion that has cropped up on the list and in bugzilla is what kind of changes can be made in Last Call, how must they be documented, and what are the relevant Process requirements. The Chairs have also discussed this amongst ourselves.

For the current Last Call, we haven't placed any hard restrictions on changes being made based on non-bugzilla feedback. However, this has led to a clash of expectations - some WG members think lots of these changes should still happen, while others expect every change to go through bugzilla. Also, if we leave the door open to arbitrary substantive changes, we risk being on an endless treadmill of Last Calls. (See <http://www.w3.org/2005/10/Process-20051014/tr.html#transition-reqs> for the W3C Process definition of "substantive change"). That would be no fun, because I'm sure we'd all like to progress HTML5 and move on to focusing on HTML.Next. 

Since the current Last Call review period is already running, but there may still be time to make some course corrections. Also, since many of our deliverables are certain to require a second Last Call, we can start thinking now about how we may want to change things for LC2, and can consider raising the bar even higher.

The Chairs do not have a firm proposal here, but we have collectedly thoughts and notes about the requirements of the W3C Process, and pragmatic considerations. We welcome input on the right approach to tightening change control in Last Call.


1) What are the hard Process requirements to document changes?

The W3C Process requires documentation of "all changes (both substantive and minor)", and requires this documentation "since the previous step" to transition to CR or later. 

http://www.w3.org/2005/10/Process-20051014/process.html#transition-reqs

>> Provide public documentation of all changes (both substantive and minor) to the technical report since the previous step. A substantive change (whether deletion, inclusion, or other modification) is one where someone could reasonably expect that making the change would invalidate an individual's review or implementation experience. Other changes (e.g., clarifications, bug fixes, editorial repairs, and minor error corrections) are minor changes.

There's also a requirement to address all issues raised since the previous step (as typically documented in a Disposition of Comments):

>> Formally address all issues raised about the document since the previous step.

Specific information that must be provided is (emphasis added):

>> The following information is important to the decision to advance a technical report and therefore must be publicly available:
>> 
>> 	• *Documentation of all changes* to the technical report (e.g., *by providing "diffs"* in addition to *summaries of substantive changes*);
>> 	• A statement that all requirements have been fulfilled or a listing of unfulfilled requirements and the rationale for advancing the document though some requirements have not been met.
>> 	• Evidence of wide review and that dependencies with other groups have been resolved;
>> 	• All *responses that formally address* issues raised by reviewers;
>> 	• All Formal Objections."


A few points:

- Per the W3C Process, both substantive and minor changes must be documented in the form of diffs, but summaries are only requested for substantive changes.
- Rationale is not specifically required for changes (though is presumably required for raised issues that must be formally addressed).
- Does CVS commit history provides adequate history at the diff level? It seems like it might.
- It doesn't seem that the CVS history is an adequate summary of substantive changes. We would need a summary of all substantive changes before entering CR.
- The W3C Process does not make clear what the reference point is for "since the last step" in the case of a CR transition. Normal practice is that the previous step is taken to be all Last Calls back to the first.
- The W3C Process has a related requirement to formally respond to every comment from the start of LC to the end of LC, and that there be a public record of this. Many Working Groups provide a Disposition of Comments document for this purpose. Many others use bugzilla for comment tracking and simply point to bugzilla as the public record of a formal response to every comment.

Tentative conclusions:

    1.A) Since the CVS history provides adequate public documentation at the "diff" level, we don't need to increase documentation requirements for minor changes.
    1.B) We do need "summary" documentation of substantive changes. This could be in the form of bugs, or it could be or it could generated manually after the fact.
    1.C) One possibly way to collect summary information is to have a group of volunteers review the commit logs and write up summaries of substantive changes that did not have a bug associated, or write up bugs for these changes retroactively.


2) What about Process requirements to do another Last Call?

It is expected that Working Groups may need to make substantive changes during a Last Call:

http://www.w3.org/2005/10/Process-20051014/process.html#last-call

>> Ideally, after a Last Call announcement, a Working Group receives only indications of support for the document, with no proposals for substantive change. In practice, Last Call announcements generate comments that sometimes result in substantive changes to a document. A Working Group should not assume that it has finished its work by virtue of issuing a Last Call announcement.

However, if a substantive change is made, the Working Group must republish as a Working Draft rather than advancing:

http://www.w3.org/2005/10/Process-20051014/process.html#return-to-wg

>> A technical report is returned to a Working Group for further work in either of the following situations:
>> 
>>     *The Working Group makes substantive changes to the technical report at any time after a Last Call announcement and prior to Publication as a Recommendation, except when the changes involve the removal of features at risk identified in a Call for Implementations. In the case of substantive changes, the Working Group must republish the technical report as a Working Draft…."

So while substantive changes are allowed and even expected, they trigger a new Last Call.

My tentative conclusions:

    2.A) Because a substantive change requires a new Last Call, it affects the whole WG. Therefore, in principle the WG should be in the loop when a substantive change is made and have a meaningful ability to say no.
    2.B) We went into LC1 expecting substantive changes. The status section even points out their likelihood. While there isn't unanimity on this, many in the WG seem to agree with the expectation. So for LC1, the choice to accept substantive changes has already in effect been made.
    2.C) Clearly we don't want to be on an endless treadmill of Last Calls.
    2.D) For LC2 it probably makes sense to set the expectation differently and require prior notice of all substantive changes.
    2.E) It may therefore make sense to have different requirements for different sorts of changes for LC1 and LC2.


3) How could we classify changes?

The following classification of changes may be helpful:

3.A) Feature additions. While the Process does not distinguish them from other substantive changes, they are not only substantive changes in themselves but also likely to generate a stream of additional comments and possible further substantive changes.
3.B) Other substantive changes (including feature removals and other kinds of non-minor changes). Per Process, these require summary documentation and a new LC.
3.C) Minor changes. Per Process, these require only diff-level documentation.

Setting aside for the moment what we do for LC1, for LC2 (or later) these standards may be sensible:

I. Feature additions should be declined except by explicit WG decision otherwise. Bugs of this type should be automatically RESOLVED LATER and changes that slip in otherwise must be reverted.
II. Other substantive changes require constructive prior notice since any that is made will require yet another LC. It's probably reasonable to require a bug ahead of the change. I'm not sure if we should give the bug a minimum lifetime to make it considered "prior". Sam previously said that he thinks a few hours are not enough.
III. Minor changes require only documentation. Thus, in principle we can do entirely without bugs or with after-the-fact bugs. The main issue would be to ensure none of these are actually substantive changes.


4) Lots of browser developers typically send their feedback to the WHATWG only with perhaps an assumption that it will make it into the W3C HTML5 spec.

There are two categories of assumptions here:

4.A) Expectations of past feedback being addressed. One possible idea is to look at the public version of Ian Hickson's input queue and put all the comments in bugzilla.

4.B) Expectations of ongoing feedback being addressed. We can change this expectation by giving sufficient prior notice that, past some date, substantive changes can only make it into the W3C spec via bugzilla.

Note: these assumptions may be bad, but the WG has done nothing to dispel them, and the spec is not well-served by dropping implementor feedback on the floor. Note that feedback from implementation experience would be valid even during CR, so we are better off addressing it sooner so we don't end up needing to drop out of CR.


5) Who's sign-off would we need for what?

- We probably need the WG to agree if we have a default "new features are rejected" policy for LC2, if we want to go that route, via a survey.

- The WG will need editors to be bought into bugs and prior notice for substantive changes during LC2, if the WG wishes to set those policies. If LC2 is short and feedback volume is low, this may not be an undue burden.