Re: Proposal: adopt a "test required" policy for spec changes

I'm in favor of adopting a policy along the lines of Web Performance's.
Requiring tests for changes to a document in CR, in particular, seems like
a no-brainer, since we ought to have enough implementation experience by
that point to make writing tests against an implementation trivial.

I'm a little more skeptical about a test-first approach to designing a
feature in the first place, though, as I see some marginal risk of locking
ourselves into a bad design just due to inertia and sunk cost. Chrome's
incubation-first strategy seems like a reasonable way of mitigating this
from a working group perspective (e.g. we'd only adopt documents (and
therefore apply a test-first policy to them)) once we were reasonably
confident that their general shape was itself reasonable.

+Philip who's been thinking about this a lot, both from Chrome's
perspective, and from the perspective of the WHATWG (where such a
policy is already
solidly in place <https://whatwg.org/working-mode>)).

-mike

On Fri, Sep 8, 2017 at 9:47 PM, Daniel Veditz <dveditz@mozilla.com> wrote:

> The W3 leadership has been emphasizing the importance of ensuring
> interoperability through Web Platform Tests. Some groups have adopted a
> policy of requiring corresponding web-platform-tests pull requests for
> before landing normative spec changes. Since interoperability is part of
> getting a spec to become a Recommendation this makes sense especially for
> specs that are in or entering CR. Should we adopt such a policy?
>
> The Web Performance group adopted the following:
> [[
> ALL normative spec changes are generally expected to have a
> corresponding pull request in web-platforms-tests, either in the form of
> new tests or modifications to existing tests, or must include the
> rationale for why test updates are not required for the proposed update.
> [...]
> ]]
> https://github.com/w3c/web-per <http://goog_1391446905>
> formance/blob/gh-pages/CONTRIB <http://goog_1391446905>UTING.md
>
> ... and the CSS Working Group adopted one last week:
> [[
> For normative changes for any specification in CR or later as well as the
> pre-CR specifications listed below, a corresponding web-platform-tests PR
> must be provided, except if testing is not practical; for other
> specifications it is usually appreciated. Typically, both PRs will be
> merged at the same time. Note that a test change that contradicts the spec
> should not be merged before the corresponding spec change.
> [...]
> ]]
> https://github.com/w3c/csswg-drafts/blob/master/CONTRIBUTING.md
>
> Good idea? Objections? Respond on list and we can talk about it on our
> next call (Sept 20).
>
> -Dan Veditz
>