Thorough Review at W3C

One of the important operational principles of W3C—and of any effective 
standards organization IMHO—is (as captured in the proposed W3C Vision¹ 
<https://www.w3.org/TR/w3c-vision/>) “Thorough Review”. I wanted to take a 
moment to dig into what that means, particularly since there's a lot of 
confusion about what constitutes “wide review”.

There are two parts to “thorough review”: wide review, and deep review. Both 
are integral to the W3C standardization process.

*Deep review* is thorough review by subject matter experts and other 
meticulous volunteers. It occurs through our working group discussions, 
particularly the contentious ones; in reviews of specific edits or of the 
whole document by the working group or external contributors; and by necessity 
through the implementation process (which is one of the reasons we require 
implementations). It is a core part of what we do at W3C, and elevates the 
quality of our specs beyond what a single company could do alone. Deep review 
ensures the technical soundness of our specs.

*Wide review* is engaging the wider impacted community and experts outside our 
own group. It helps us ensure the technology is fit for purpose and integrates 
well with the broader (technical and human) ecosystem. A good wide review 
requires some thoughtfulness about how to do it: Who is the impacted 
community, and how can we engage them? Who are additional experts outside our 
group whose knowledge and experience might be valuable in reviewing this, and 
how can we reach them?

Wide review is often conflated with horizontal review² 
<https://www.w3.org/Guide/documentreview/#how_to_get_horizontal_review>, which 
is a mechanism we use at W3C as /part/ of wide review. Horizontal review 
engages W3C's in-house experts on certain identified horizontal concerns. It 
is comparatively easy: you follow the checklist, and you're done. You don't 
have to figure out how to do it. But it's not enough, because wide review is 
both internal /and external/ to W3C. Our stakeholders include the broader 
industry and the public, so wide review requires /outreach/.

In the CSSWG, wide review includes posting about our work on our blog and 
social media to invite attention and feedback. It includes the DevRel 
(Developer Relations) efforts of our implementers, who talk up the work their 
team is doing to implement our specs, inviting experimentation and bug reports 
on their latest features. It includes producing prototype builds and 
encouraging users to try them out and send feedback. And it includes reaching 
out to our Horizontal Working Groups, to Unicode, or to other groups and 
individuals with relevant expertise for the spec in question.

Working on the Process Document is different. Here, wide review includes 
reaching out to the AC and the Chairs about proposed upcoming changes, 
discussing and working with the Team on designing and preparing to implement 
those changes, and checking in with PSIG and W3C's legal counsel to make sure 
we didn't break any legal requirements. Social media posts are not so necessary.

In all cases, however, wide review requires us to put the changes in view of 
the wider community and invite comment. This is why we publish our specs as 
drafts for review, and not just once they are final. It ties in with our 
operational principles of openness, transparency, and diversity, because these 
are what enable an effective wide review.

Effective wide review for EPUB is different from effective wide review for CSS 
or DID. That's why wide review is not a checklist: each technology needs to 
design its own wide review procedures. Also, while new proposals need to cast 
a broader net to ensure nothing was missed, wide review for changes should be 
tailored to the change. Fixing errors in a layout algorithm only really needs 
attention by implementers and the Working Group. But if the change impacts 
typesetting, even though we likely don't need to bother Security or the TAG, 
it might be appropriate to get Internationalization involved. If we're 
substantially altering an API design, we'll want to give authors an 
opportunity to comment. Etc. A good wide review process is tailored to what is 
being reviewed, both the matter and the scope.

The Team plays an important role in wide reviews: Team Contacts in coaching 
our Groups on doing it well, and the Team's transition request verifier in 
ensuring an appropriate level of review was accomplished.

Wide review is hard to do well. It's a lot of work to engage people outside 
your own team, a lot of work to process their feedback. It requires thought 
and consideration to figure out how to do it effectively for any particular 
thing. Even I don't always do a good job. But wide review is an important part 
of what makes the W3C standardization process better.

I'm writing this post for several reasons:

  * To encourage us to think about our review mechanisms: *wide* (broader
    community) and *deep* (detailed expertise), *internal* (within our own
    groups) and *external* (reaching out beyond our teams), *explicit*
    (intentional requests for review) and *implicit* (processes that build in
    an effective side-effect of review).
  * To acknowledge the important role the Team plays here, and encourage more
    effective use of the Team in this area.
  * To more explicitly adopt the concept of “wide review” outside of our work
    on technical reports, because so many of our other projects have and can
    benefit from it.

Thanks~
~fantasai

¹ https://www.w3.org/TR/w3c-vision/
² https://www.w3.org/Guide/documentreview/#how_to_get_horizontal_review

Note: This message is CCed to www-archive@w3.org which is publicly archived 
<https://lists.w3.org/Archives/Public/www-archive/>.