Tentative tests.

Following up on "when automerging of tests goes awry
<https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg25>"
and "Dealing with prove-it-then-spec-it situations in web-platform-tests
<https://groups.google.com/a/chromium.org/d/msg/blink-dev/LDc3RgrrldU/a1Gn0ld9EQAJ>"
(and CCing various folks who were involved in those conversations). Based
on those conversations, it sounds like there's something approaching
consensus that it would be good to support "tentative" tests in WPT that
run ahead of the relevant specification. This allows us to gather
implementation experience to inform our decisions about the "right" answer
to spec questions, and allows us to share that experience easily with each
other.

In order to ensure that we're on the same page about the status of these
tests, I'd propose that when uploading tentative tests, developers choose
one of the following identification mechanisms:

1.  Put the tests in a new file with a `-tentative.*` suffix (e.g.
`//content-security-policy/script-src/new-test-tentative.sub.html`).

2.  Put the tests in a `tentative` subdirectory of an existing suite (e.g.
`//content-security-policy/tentative/new-script-src-test.sub.html`).

These both seem fairly unambiguous, and I see good use-cases for each.

WDYT?

-mike

Received on Tuesday, 23 May 2017 13:11:22 UTC