Encouraging/Ensuring Wide Review

I have tried to capture and put in time order the most relevant comments on the removal of "Last Call" as a maturity level. I would note that the discussion is mostly about informing potential reviewers what they are supposed to be doing (with any given Working Draft). I am adding my own comments to the discussion at the end.

David Singer singer@apple.com<mailto:singer@apple.com>, Fri 10/18/2013 3:51 PM

"we have a document, please give us public comment"

"OK, we have addressed the public and other comments, now implement and tell us what we missed"



do seem to be pretty separate steps.  how does the new process envisage them?

Charles McCathie Nevile chaals@yandex-team.ru<mailto:chaals@yandex-team.ru>, Sat 10/26/2013 1:36 AM

As the responsibility of the Working Group to determine how to handle. It seems that in most successful cases there is in fact implementation going on at a very early stage, often before the public has even had a chance to comment.

David Singer singer@apple.com<mailto:singer@apple.com>, Fri 10/25/2013 6:26 PM

[re: WG handling] That's fine.  Just give people a name they can use (a consistent name) if they wish.



[re: early implementations] It's not pretty to tell the public "oh, thanks for your feedback, but there are too many implementations to change now"...  :-(



Charles McCathie Nevile chaals@yandex-team.ru<mailto:chaals@yandex-team.ru>, Mon 11/4/2013 11:28 PM

[re: WG handling] How about "getting ready for LCCR"?



There are currently groups that have a "pre-last-call call for review".

This is OK as a practice, but I don't think putting the name pre-last-call in the process would be an improvement in any way...



[re: early implementations] But this is not a new problem. Groups have been trying that at least since I first filed a comment on a spec 15-odd years ago, and have subsequently been prepared to revise their work based on clear arguments. At the same time, implementation and deployment actually does matter in the real world, so without automatically declaring that requiring changes to implementations is bad, it makes sense to take into account the status of deployment. Indeed, under the proposal groups SHOULD document known implementation, for that reason inter alia.



David Singer singer@apple.com<mailto:singer@apple.com>, Tue 11/5/2013 2:27 AM

[re: WG handling] a name for the *document* that they believe is their candidate.

Charles McCathie Nevile chaals@yandex-team.ru<mailto:chaals@yandex-team.ru>, Tue 11/5/2013 2:54 AM
[re: WG handling] [SNIP]

I'm unclear why a working draft needs a particular name, and sceptical that adding such names to the process is helpful.



The status section is there to clarify exactly where a document is up to, when it is between the clearly defined stages. And Working Groups are meant to be freer to determine how they want to handle the progression.

Against that, I'd rather not define lots of stage names.


If there are clear requests based on multiple groups doing the same thing I think we should reconsider this question. I haven't seen that to date.

David Singer singer@apple.com<mailto:singer@apple.com>, Wed 11/6/2013 9:58 PM

[re: WG handling] Because there are bewildering numbers of WDs from groups, most of them unreadable or not really ready for comment.  Something that says "ok guys. this might be half-readable" is a Nice Sign.



"Draft Proposal" or "Proposed Draft" would be fine, for example.



Michael Champion (MS OPEN TECH) Michael.Champion@microsoft.com<mailto:Michael.Champion@microsoft.com>, Wed 11/6/2013 10:49 PM

David raises a point that I don't recall the AB thinking very hard about:  People OUTSIDE the WGs need clear guidance about when to seriously review drafts.  We've tended to think that since WGs operate in public, the people outside the WG who care about a spec can read it,so they will read it as it evolves.  David reminds us that the noise level is so high unless one follows closely, it's not clear when it's REALLY ready for a wide review round.



What we did consider are a couple of points:

- The LC and CR signals in the current process tend to happen these days after specs are widely implemented and what might sound like constructive suggestions (e.g., "the names aren't very intuitive") are made too late to be helpful.

- The spirit of all these proposed reforms is to allow WGs and their chairs to make context-appropriate decisions about when and how to do things like ask for wide review.  Likewise the process forces seemingly arbitrary transitions back to WD and then forward to LC or CR, and those transitions aren't necessarily useful signals to non-WG members.  I think we understand that this does put more of a responsibility on chairs, but hopefully more freedom for them to customize and optimize the workings of a group to meet its real constraints rather than having them dictated by the One True Process.



Tobie Langel tobie@fb.com<mailto:tobie@fb.com>, Thu 11/7/2013 5:40 AM

[re: LC and CR signals] It is absolutely critical to get developer eyeballs looking at the specs before it's too late to change the APIs. Anything that helps with this is an important step forward.



Marcos Caceres w3c@marcosc.com<mailto:w3c@marcosc.com>, Thu 11/7/2013 6:03 AM

[re: LC and CR signals] Agree. It's also never too early to get input into a spec from anyone. Taking away signals that allow people to wait longer to provide feedback would actually be a good thing. LC or CR is _waaaaayyy_ too late to be providing feedback.



Sylvain Galineau galineau@adobe.com<mailto:galineau@adobe.com>, Thu 11/7/2013 5:38 PM

[re: LC and CR signals]It's too late because LC assumes most substantive feedback has already been handled. A few weeks of LC only makes sense if you assume wide review to have already happened. So I'm not sure we're debating a new issue here.

Most web developers I know never really understood what Last Call means or what kind of a time window it involves. The new process does not fix this; I do not think the previous one did either.



Charles McCathie Nevile chaals@yandex-team.ru<mailto:chaals@yandex-team.ru>, Fri 11/8/2013 8:14 AM

[re: Galineau, "too late"] Indeed. So the new process makes wide review a requirement before you get to LCCR.



(And yeah, the name is horrid. Any consensus on what it should be? It seems at a glance that Candidate Recommendation is the popular choice).



[re: Galineau, "process does not fix"] It sets a little bit of expectation, but it errs on the side of giving responsibility to Working Groups, rather than giving them administrative hoops to jump.



> I do not think the previous one did either.



A little, but not much.



Ultimately, the process can mandate what it wants, but getting the right eyeballs on the spec is the job of the Working Group and nothing written in the requirements is more than encouragement to do it right.



Michael Champion (MS OPEN TECH) Michael.Champion@microsoft.com<mailto:Michael.Champion@microsoft.com>, Fri 11/8/2013 9:56 AM

[re: WG responsibility] That's a great short summary of what the AB and the Process CG are trying to do here.  But I'm hearing feedback suggesting that we need to think a bit harder about the "signaling" role of the process steps we're proposing to eliminate.  Maybe the process document should say something about the mechanism by which WGs and chairs should signal a need for outside review even though we're decoupling the signals from the "administrative hoops."



The process has to be flexible.  There are probably going to be traditional WGs that start with nothing but a charter, others that start with multiple input specs with the same scope but different syntax/semantics, and probably more and more that start with  fairly mature CG Final Report, member contribution, snapshot of a "living" spec, etc. Maybe we can handle all these scenarios better in the revised process document, but we can't pretend that all WDs are equally open to substantial revision by external reviewers just because they are at the same stage in the formal process. That's the bug we're trying to fix here.



On a related matter, we might want to tweak the draft process to give clearer guidance about how to prevent the most frustrating "administrative hoop" that I hear about:  Going backwards in the process in order to meet late-breaking requirements or remove unimplemented features.Maybe the new process should encourage WGs to move forward to Rec with the features that are really interoperable, dropping other features whether or not they were marked "at risk" and moving them to the queue for v.next.  That's not viable when there is a 5-10 year timeline between versions of a Rec, but it's business as usual in "agile" software development approaches we're trying to learn from.  Yes those situations probably require a louder "you need to review this, it's not what we said we would do" signal from WGs.  The AB has proposed to treat this as the responsibility of WGs to execute properly rather than as inexplicable bureaucratic minutia that WGs have to do "just because".



So this is sort of a rambling way to ask the AC:  Should the new process say more about the conditions and mechanisms of a "call for wide review" or should we leave this as an implementation detail for WGs to figure out how to do properly given the constraint that they ultimately need to convince the Director that the wide review has happened?



Wayne Carr wayne.carr@linux.intel.com<mailto:wayne.carr@linux.intel.com>, Fri 11/8/2013 12:49 PM

In the past we tied things to formal publication stages that don't need to be tied to a particular stage of the whole draft.



Any TR publication, like a heartbeat draft, could have a section right after status that was "Call for Review".  that section would point out sections and particular issues the WG particularly wants feedback on.

Individual sections of the draft or issues for review could be noted throughout the draft.



So, it wouldn't need to be a review that the whole draft is ready for LCCR.  It could be the WG thinks a particular section is done and indicate the WG is considering labeling it as stable.



There could be a public call for review mail list for all W3C specs where publications of TR drafts with calls for review were announced

(not a discussion list - a low volume announcement list).   So instead of

trying to track WG mail lists, someone outside the WG who wanted to know when to review, could subscribe to that list.  That would also provide a record of the WGs effort to get wide review.



There could also be a similar notification section about which sections are stable in drafts (have been through a call for review and issues are resolved to the extent the WG is going to resolve them).  With notification when sections are marked as stable by a WG after successful reviews to the call for reviews list (as an outcome of the review).



Steve Zilles szilles@adobe.com<mailto:szilles@adobe.com>, Mon 11/11/2013, 3:43AM

In addition to Wayne's proposals, there are separate threads [1, 2] that argue for restructuring the Status Section. There are two parts to this thread, one please do not make the Process say what must be in the "Status" section (because that makes it difficult to do a (mostly editorial) restructuring of the material into more relevant sections), and what information should be required by the Process.



[1] http://lists.w3.org/Archives/Public/public-w3process/2013Oct/0105.html

[2] http://lists.w3.org/Archives/Public/public-w3process/2013Nov/0020.html



Combining these the above discussion with the Status Section  discussion, I would propose that

1.        the Process Document should require certain classes of information at the beginning of the document, but should not necessarily specify in which section(s) that information should appear. The section organization can be left to Pub Rules

2.       There be (ideally IMO) a section with a name like, "Review Topics" (or "Call for Review") that highlights sections of the document that need review. This is not intended to replace (although it may point to) the "Change List" section (which should be more complete) that is toward the end of the document. It is intended to help people that are not following the day to day activity of a document or working group have some idea of what to do with that document when it arrives. Another way of saying this is that there will never be a "Last Call"; there will always be something to review in every new Working Draft (or a statement that there has been no change since the last Working Draft) until the document becomes a Recommendation.

3.       Besides a "Review Topics" section, a list of Changes since the last Working Draft, it would also be useful if sections (and sub-sections where relevant) have some annotations to indicate information about that section, such as when last edited, how many implementations, how many tests and they are passed (inter-operably). CSS currently does some of this for some of its Editors Drafts; e.g. CSS Regions
http://dev.w3.org/csswg/css-regions/

4.       I am ambivalent as to whether there should be a separate "Drafts Announcement List". If every Working Draft has some Review needs, then simply announcing the draft should suffice. I am concerned that a "Draft Announcement List" would have so much traffic that it would become useless unless is was structured so that users could simply define filters or other active agents to select items that they feel that should look at in more detail.



The above proposal means a fairly significant change to the way Reviewers work with W3C documents in development. This change (as was noted above) comes from encouraging an Agile Specification Development approach. With small(er) components being developed in detail and likely implemented and tested. It is necessary that Reviews become incremental (not at the Editors draft level) but at the Working Draft level. Each of these Reviews should be simpler than a full document "Last Call" review, but they need to happen more frequently. This may mean a greater burden on the Reviewers over all, but should also mean that their comments receive better consideration and incorporation. And, it is hoped that using a Agile Specification Development approach, that requirements will be met in a more timely fashion and in roughly priority order.



Steve Zilles