W3C home > Mailing lists > Public > public-credentials@w3.org > September 2016

Rebalancing How the Web is Built

From: Manu Sporny <msporny@digitalbazaar.com>
Date: Sun, 11 Sep 2016 11:20:44 -0400
To: Credentials Community Group <public-credentials@w3.org>
Message-ID: <57D5764C.8020301@digitalbazaar.com>
Hi all,

Here's an article reflecting on some of the challenges we've faced
getting work started at W3C on the timeline that we desire. It's a
complex problem and this blog post attempts to deconstruct why this
dynamic at W3C exists. Here's a summary of the post:

The World Wide Web Consortium members standardize technology for the
next generation Web. It’s arguable that the way this process works does
not provide a clear path to progress work and favors large
organizations. This blog post explains why we ended up here and how we
could make the process more fair and predictable.

http://manu.sporny.org/2016/rebalancing/

The contents of the blog post can be found below for W3C archival purposes:

---------------------
Rebalancing How the Web is Built

   Since the early 1990s, the World Wide Web Consortium (W3C) has
   been the organization where most of the next generation Web has
   been standardized. It is where many of the important decisions
   around HTML, CSS, and Browser APIs are made. It is also where many
   emerging non-browser related technologies like Linked Data,
   Blockchain, Automotive Web, and Web of Things are incubated.

   The consortium has always tried to balance the needs of large
   corporations with the needs of the general public to varying
   degrees of success. More recently, a number of the smaller
   organizations at W3C have noted how arduous the process of
   starting new work at W3C has become while the [10]behavior of
   large organizations continues to cause heartburn. As a result,
   some of us are concerned that the process is going to further tilt
   the playing field toward large organizations and unnecessarily
   slow the rate of innovation at the heart of the Web.

   In this blog post I’m going to try to explain how creating new
   technology for the core of the Web works. I’ll also suggest some
   changes that W3C could make to make the process more predictable
   and hopefully more fair to all participants.

> From There to Here

   For most of its existence, the W3C has had a process for
   developing next generation technology for the [11]Web Platform
   that goes something like this:

    1. Wait for a group of people to identify a problem in the Web
       Platform and propose how adding specific new features would
       address the problem.
    2. Create a Working Group to standardize the features in the Web
       Platform that would solve the identified problem.
    3. Release an “official” technical specification under a patent
       and royalty-free license that is used by Web developers to
       varying degrees of success.

   It takes anywhere from 3-6 years for a technology to get through
   the W3C process. The formation and operation of a Working Group
   tends to be where the costs are fairly high in terms of W3C staff
   resources, which means the cost of failure is also relatively high
   for the organization as a whole. This has led to changes to the
   process that have raised the bar, in a negative way, on what is
   necessary to start a Working Group. The playing field has been
   tilted to the point that some of us are concerned that a handful
   of large organizations now have a tremendous amount of influence
   on where the Web is going while large coalitions of smaller
   organizations or the general public struggle to have their
   concerns addressed in the Web Platform.

   Recently, W3C has [12]undergone a reorganization to make it more
   responsive to the needs of the organizations and people that use
   the Web. The reorganization splits the organization into multiple
   functional groups: Strategy, Project Management, Architecture and
   Technology, Global Participation, Member Satisfaction, Business
   Development, etc. While the reorganization feels like a move in
   the right direction, it doesn’t seem to address how difficult it
   is to start and shepherd work through the W3C. Let’s analyze why
   that is.

Why Shepherding Work Through the W3C is Onerous

   It is claimed that one of the primary bottlenecks at W3C when it
   comes to starting a new Working Group is allocating W3C staff
   member time to the project. The argument goes that there are just
   not enough W3C staffers for the workload of the organization. This
   is because W3C exists on a thin gruel of funding from membership
   fees (as well as other disparate sources) and hiring W3C staff to
   help support new work is financially burdensome. We have limited
   resources, so we must only pursue work that has the very highest
   likelihood of success. This means that parts of the Web Platform
   ecosystem develop far more slowly than they should.

   This dynamic has prompted a number of the larger organizations at
   W3C to push back on new charters citing this bottleneck. This is a
   misguided strategy. We shouldn’t be focused on eliminating as much
   risk as possible, we should be focused on reducing the cost of
   making mistakes.

   As a result of the focus on eliminating as much risk of failure as
   possible, some of these large organizations have started
   requesting new requirements for starting work. Some examples
   include requirements like complete technical specifications,
   “significant” deployments to customers, and large vendor support.
   These new requirements are intended to increase the likelihood of
   success and slow the rate of new work at W3C. While slowing the
   rate of new work may address the staffing issue, it also creates a
   situation where the W3C can’t keep up with the required amount of
   standardization needed for the Web Platform to stay relevant. It
   also creates a bias in favor of large organizations.

   We have seen these new requirements ignored when it suits a large
   organization to do so (e.g. Web Payments and Web Application
   Security). If a large organization can see how their company
   financially benefits from an added feature to the Web Platform
   they seem to have a much easier time getting work started at the
   W3C as the “starting new work” requirements are not applied to
   them in the same way as they are to the smaller organizations at
   W3C.

   While I won’t go into all of them here, there are more double
   standards, no pun intended, at play when transitioning work to a
   Working Group at W3C. This dynamic results in significant
   frustration from the smaller organizations at W3C when the goal
   posts for starting new work keep changing based on the current
   desires of the larger organizations.

   Regardless of whether or not this is a staffing issue, a
   technology maturity issue, or one of the other new requirements at
   play at W3C, one thing is certain. The list of requirements for
   transitioning work from an experimental technology to a Working
   Group at W3C is not clear and seems to favor larger organizations.
   It is a source of extreme frustration for those of us that are
   trying to help build the next generation Web and are NOT working
   for a multi-billion dollar multinational corporation.

The W3C Working Group Formation Checklist

   One remedy for the aforementioned problems is to employ a simple
   but detailed checklist. This is one of the tools that is currently
   missing for W3C members.

   Some have argued that this lack of a clear checklist is by design.
   Some argue that the formation of every new Working Group is unique
   and requires discussion and debate. Having participated in
   starting multiple groups at W3C over the past decade, I disagree.
   The details differ, but the general topics that are debated remain
   fairly consistent. The data that you need to support the creation
   of a Working Group tend to be the same.

   Here is the checklist that I’ve put together over the years after
   much trial and error at W3C. The purpose of the checklist is to
   gather data so that your community can prove that it has done its
   due diligence to the W3C membership when requesting the creation
   of a Working Group.

    1. Clearly identify and articulate a problem statement. Do this
       by sending out a survey to all organizations that you believe
       may benefit from a standard. You will need at least 35
       organizations to become actively involved. I typically end up
       having to contact close to 100 organizations to get this core
       group formed. Example: [13]Digital Offers Problem Statement
       Survey.
    2. Create a W3C Community Group around a broad version of the
       problem statement and invite organizations to join the group.
       Organize weekly calls to coordinate activity. Example:
       [14]Credentials Community Group
    3. Gather use cases that these organizations want to support.
       Specifically, gather use cases that are currently impossible
       to achieve. This should be in the form of another survey where
       you ask each organization about their most important use case.
       You will eventually process this list down to 3-5 core use
       cases. Example: [15]Verifiable Claims Use Cases Survey and
       [16]Verifiable Claims Use Cases.
    4. Perform a gap analysis. Identify capabilities that are missing
       from the Web Platform and explain why those capabilities can
       address some of the use cases outlined in the previous step.
    5. Produce a proposed architecture and technical specification
       that addresses part or all of the problem statement. If
       possible, get multiple implementations of the technical
       specification and deploy it at a few customer sites. Example:
       [17]Verifiable Claims Architecture Document
    6. Create a draft charter for a Working Group that will take the
       technical specification through the W3C standardization
       process. The charter time frame should be for no more than 24
       months: 6 months to spin up, 12 months to finalize the specs,
       6 months to complete interoperability testing. Example:
       [18]Verifiable Claims Working Group Draft Charter
    7. Create an executive summary for W3C member companies.
       Summarize the work that the group has done. At this point you
       have far more content than W3C member company representatives
       have the time to read to determine if they want to join the
       initiative. Ease their decision making process by providing a
       summary. Example: [19]Verifiable Claims Executive Summary for
       W3C Members
    8. Measure buy-in for the proposed Working Group Charter and
       technical deliverables. The best way to do this is via another
       survey that should be distributed to any organization that
       participated in step #1, any organization that has joined the
       work since then, and any W3C member organization that may be
       impacted by the work. Example: [20]Demonstration of Support
       for Verifiable Claims Working Group Charter Survey

   I don’t expect much in the list above to be controversial. There
   are templates that the W3C membership should create for the
   surveys and reports above. Having this W3C Working Group Formation
   Checklist available will help organizations navigate the often
   confusing W3C Process.

Reducing Reliance on W3C Staff

   The checklist above makes the process of going from an
   experimental technology to a W3C Working Group more clear. The
   checklist, however, does not alleviate the W3C staffing shortage.
   In fact, if the way W3C staff is utilized does not also change, it
   might make the current situation worse.

   The W3C staff play a very important role in that they help
   organizations navigate how to get things done via the W3C Process.
   They help build consensus before and during a Working Group
   activity at W3C. This can be a double-edged sword. If there is a
   W3C staff member available to help, they can be a fantastic
   champion for a group’s work. If there isn’t, and you’ve never
   progressed work yourself through W3C, you will find yourself in
   the unfavorable position of not knowing how to proceed. Many of
   the staff members also have their own way of navigating the W3C
   process and many of them have to repeat themselves when new work
   is started. In short, there is a lot of engineering churn and
   tribal knowledge at W3C; this is true of most standardization
   bodies.

   The unfortunate truth is that building out the Web platform is
   currently being restricted due to the way that we start and
   progress new work at W3C. The W3C membership relies on the W3C
   staff to its detriment. The W3C staff are incredibly helpful, but
   there aren’t enough of them to support building out all of the new
   functionality needed for the Web, and this is ultimately bad for
   the Web.

   In order to address the staffing shortage, it is proposed that we
   shift much of their work from executing upon the proposed
   checklist, which is more or less what they do today, to verifying
   that the checklists are being processed correctly. This is
   effectively a quality control check on the work that groups are
   doing as they work their way through the proposed checklist. This
   offloads a significant chunk of W3C staff time to the groups that
   want to see their work get traction while giving those groups
   clear goals to achieve.

Blocking Work at W3C

   W3C members currently have the ability to stop work that has
   ticked all of the necessary boxes in the checklist mentioned
   above. At present, large organizations tend to not become involved
   with work until a vote for the Working Group is circulated by W3C
   staff. Some members then suggest that they may vote against the
   work if their viewpoint isn’t taken into account even though they
   have not participated in the work to date. Some call this a part
   of the process, but it is uselessly frustrating to those that have
   spent years building consensus around a Working Group proposal
   only to have a large W3C member company respond with a “Why wasn’t
   I consulted” retort.

   So, now for the controversial bit:

   A W3C Community Group’s work should automatically transition to a
   W3C Working Group if a significant coalition of companies, say
   around 35 of them, have ticked all of the boxes in a predefined
   W3C Working Group Formation checklist.

   This checklist should include the creation of two interoperable
   implementations and a test suite. In other words, it should meet
   all of the minimum bars for a Working Group’s success per the W3C
   Process. Making this change achieves the following:

    1. It clearly defines how experimental technology progresses to
       being part of the Web platform so that organizations can
       allocate resources appropriately.
    2. It reduces the workload on W3C staff members, which is a very
       limited resource.
    3. It prevents large organizations from blocking work at W3C
       while addressing their concerns around technological maturity.

   In the end, the checklist for starting work at W3C could look
   something like this:

    1. Clearly identify and articulate a problem statement.
    2. Create a W3C Community Group.
    3. Gather and document use cases.
    4. Perform a gap analysis.
    5. Produce a proposed architecture and technical specification.
    6. Produce two implementations and a test suite.
    7. Create a draft charter for a Working Group.
    8. Create an executive summary for W3C member companies.
    9. Survey organizations and demonstrate that at least 35 of them
       are supportive of the work and technical direction.

   Each step above would have document templates that W3C members can
   use as a starting point. None of the steps above require W3C staff
   resources and if all steps are completed, the formation of a
   Working Group is the natural outcome and should not be blocked by
   the membership.

   This checklist approach may be seen as too constraining for some,
   and that’s why it’s voluntary. Some organizations may feel that
   they do not need to produce all of the work above to get a Working
   Group and for those organizations, they can choose to ignore the
   checklist. Initiatives not using the checklist should expect push
   back if they choose to not answer some of the questions that the
   checklist covers.

Rebalancing How the Web is Built

   The current process for developing next generation standards for
   the Web is too unpredictable and too constrained by limited W3C
   resources. There are groups of small organizations that want to
   help create these next generation standards; we have to empower
   those groups with a clear path to standardization. We must not
   allow organizations to block work if the champions of the work
   have met the requirements in the checklist. Doing this will free
   up W3C staffing resources to ensure that the Web Platform is
   advancing at a natural and rapid rate.

   A W3C Working Group formation checklist would help make the
   process more predictable, require less W3C staff time to execute,
   and provide a smoother path from the inception of an idea, to
   implementation, to standardization of a technology for the Web
   platform. This would be great for the Web.

References

  10. https://manu.sporny.org/2016/browser-api-incubation-antipattern/
  11. https://www.w3.org/wiki/Open_Web_Platform
  12. https://www.w3.org/2016/08/2016-reorg.html
  13.
https://docs.google.com/forms/d/e/1FAIpQLScVDxYlBTWchzK3kThdRTn0nulipPy6Hb5qLmRDLOdBtioFfQ/viewform
  14. https://www.w3.org/community/credentials/
  15.
https://docs.google.com/forms/d/e/1FAIpQLSdQJWx8eMRkoTwA2TIWbigTipZ2lqZ_UPFzJbY8IvqmVAHr6A/viewform
  16. https://w3c.github.io/webpayments-ig/VCTF/use-cases/
  17. http://w3c.github.io/webpayments-ig/VCTF/architecture/
  18. http://w3c.github.io/webpayments-ig/VCTF/charter/rc-2.html
  19. http://w3c.github.io/webpayments-ig/VCTF/primer/
  20. http://w3c.github.io/webpayments-ig/VCTF/support/

-- manu

-- 
Manu Sporny (skype: msporny, twitter: manusporny, G+: +Manu Sporny)
Founder/CEO - Digital Bazaar, Inc.
blog: Rebalancing How the Web is Built
http://manu.sporny.org/2016/rebalancing/
Received on Sunday, 11 September 2016 15:21:14 UTC

This archive was generated by hypermail 2.3.1 : Sunday, 11 September 2016 15:21:14 UTC