RE: Adopting a dual spec/testing process for webperf specs

Late sound off.. but this SGTM as well.

We have some debt to pay off, but this will pay dividends in the long run!

-Todd

From: Ilya Grigorik [mailto:igrigorik@google.com]
Sent: Friday, March 31, 2017 2:48 PM
To: Yoav Weiss <yoav@yoav.ws>
Cc: Philip Jägenstedt <foolip@chromium.org>; Rick Byers <rbyers@google.com>; Philippe Le Hégaret <plh@w3.org>; public-web-perf@w3.org
Subject: Re: Adopting a dual spec/testing process for webperf specs

sgtm.

On Thu, Mar 30, 2017 at 12:45 PM, Yoav Weiss <yoav@yoav.ws<mailto:yoav@yoav.ws>> wrote:
I support that proposal as well.

On Wed, Mar 29, 2017 at 11:33 AM Philip Jägenstedt <foolip@chromium.org<mailto:foolip@chromium.org>> wrote:
I'd love to see this as well, everywhere! It was a genuine surprise to us when adopting it for HTML how well it worked out, and now it's hard to imagine going back. It does require a strong cooperation between spec editor and implementers, if there isn't a sense of shared responsibility, then it'll not be as great I think.

On Wed, Mar 29, 2017 at 11:35 PM Rick Byers <rbyers@google.com<mailto:rbyers@google.com>> wrote:
I'd (unsurprisingly) love to see this!

Note that when new features ship in blink we're now asking people<https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/web-platform-tests%7Csort:relevance/blink-dev/leQDM4nhGHA/Gy5LHezwCAAJ> to explain any cases where web exposed behavior does not have web-platform-tests.  So we expect writing web-platform-tests to increasingly be part of any blink implementation.  Hopefully that means this is less of a burden on spec editors than it might first seem (and ultimately less of a burden on engine developers since we get to share most of this work across companies and do less engine-specific test work).

Rick

On Wed, Mar 29, 2017 at 10:24 AM, Philippe Le Hégaret <plh@w3.org<mailto:plh@w3.org>> wrote:
Our specifications and our tests are out of sync. Most often, the tests are behind (eg Beacon) and sometimes, the tests are ahead (eg User Timing). This is costing us dearly in the long run imho (eg TAO, user-timing/mark/measure).

I'd like to propose that the Working Group adopts a dual spec/testing process, similar to the one applied in the pointer events working group [1] and the whatwg [2]:

[[
Normative spec changes are generally expected to have a corresponding pull request in web-platform-test. Outstanding test work is tracked via issues in this repository and issues generally remain open until both spec and test changes land. If one PR is approved but the other needs more work, add the 'do not merge yet' label or, in web-platform-tests, the 'status:needs-spec-decision' label.
]]

wdyt?

Philippe



[1] https://github.com/w3c/pointerevents/blob/gh-pages/README.markdown

[2] https://github.com/whatwg/meta/blob/master/TEAM.md