Re: Tentative tests.

On 23/05/17 14:10, Mike West wrote:
> Following up on "when automerging of tests goes awry 
> <https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg25>" 
> and "Dealing with prove-it-then-spec-it situations in web-platform-tests 
> <https://groups.google.com/a/chromium.org/d/msg/blink-dev/LDc3RgrrldU/a1Gn0ld9EQAJ>" 
> (and CCing various folks who were involved in those conversations). 
> Based on those conversations, it sounds like there's something 
> approaching consensus that it would be good to support "tentative" tests 
> in WPT that run ahead of the relevant specification. This allows us to 
> gather implementation experience to inform our decisions about the 
> "right" answer to spec questions, and allows us to share that experience 
> easily with each other.
> 
> In order to ensure that we're on the same page about the status of these 
> tests, I'd propose that when uploading tentative tests, developers 
> choose one of the following identification mechanisms:
> 
> 1.  Put the tests in a new file with a `-tentative.*` suffix (e.g. 
> `//content-security-policy/script-src/new-test-tentative.sub.html`).
> 
> 2.  Put the tests in a `tentative` subdirectory of an existing suite 
> (e.g. `//content-security-policy/tentative/new-script-src-test.sub.html`).
> 
> These both seem fairly unambiguous, and I see good use-cases for each.
> 
> WDYT?


Seems reasonable. Tentative seems hard to spell :) Do you anticipate 
this interacting with the manifest/runner in any way (i.e. should we be 
adding the tentative status as test metadata so that e.g. one could skip 
all tentative tests)?

Received on Tuesday, 23 May 2017 13:23:41 UTC