Re: Tentative tests.

On 23/05/17 14:10, Mike West wrote:
> Following up on "when automerging of tests goes awry 
> <https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg25>" 
> and "Dealing with prove-it-then-spec-it situations in web-platform-tests 
> <https://groups.google.com/a/chromium.org/d/msg/blink-dev/LDc3RgrrldU/a1Gn0ld9EQAJ>" 
> (and CCing various folks who were involved in those conversations). 
> Based on those conversations, it sounds like there's something 
> approaching consensus that it would be good to support "tentative" tests 
> in WPT that run ahead of the relevant specification. This allows us to 
> gather implementation experience to inform our decisions about the 
> "right" answer to spec questions, and allows us to share that experience 
> easily with each other.
> 
> In order to ensure that we're on the same page about the status of these 
> tests, I'd propose that when uploading tentative tests, developers 
> choose one of the following identification mechanisms:
> 
> 1.  Put the tests in a new file with a `-tentative.*` suffix (e.g. 
> `//content-security-policy/script-src/new-test-tentative.sub.html`).
> 
> 2.  Put the tests in a `tentative` subdirectory of an existing suite 
> (e.g. `//content-security-policy/tentative/new-script-src-test.sub.html`).
> 
> These both seem fairly unambiguous, and I see good use-cases for each.
> 
> WDYT?


Seems reasonable. Tentative seems hard to spell :) Do you anticipate 
this interacting with the manifest/runner in any way (i.e. should we be 
adding the tentative status as test metadata so that e.g. one could skip 
all tentative tests)?