W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2017

Re: Tentative tests.

From: James Graham <james@hoppipolla.co.uk>
Date: Tue, 23 May 2017 14:23:12 +0100
To: public-test-infra@w3.org
Message-ID: <f602ed9d-cdce-35ab-bdeb-88554425af2c@hoppipolla.co.uk>
On 23/05/17 14:10, Mike West wrote:
> Following up on "when automerging of tests goes awry 
> <https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg25>" 
> and "Dealing with prove-it-then-spec-it situations in web-platform-tests 
> <https://groups.google.com/a/chromium.org/d/msg/blink-dev/LDc3RgrrldU/a1Gn0ld9EQAJ>" 
> (and CCing various folks who were involved in those conversations). 
> Based on those conversations, it sounds like there's something 
> approaching consensus that it would be good to support "tentative" tests 
> in WPT that run ahead of the relevant specification. This allows us to 
> gather implementation experience to inform our decisions about the 
> "right" answer to spec questions, and allows us to share that experience 
> easily with each other.
> In order to ensure that we're on the same page about the status of these 
> tests, I'd propose that when uploading tentative tests, developers 
> choose one of the following identification mechanisms:
> 1.  Put the tests in a new file with a `-tentative.*` suffix (e.g. 
> `//content-security-policy/script-src/new-test-tentative.sub.html`).
> 2.  Put the tests in a `tentative` subdirectory of an existing suite 
> (e.g. `//content-security-policy/tentative/new-script-src-test.sub.html`).
> These both seem fairly unambiguous, and I see good use-cases for each.

Seems reasonable. Tentative seems hard to spell :) Do you anticipate 
this interacting with the manifest/runner in any way (i.e. should we be 
adding the tentative status as test metadata so that e.g. one could skip 
all tentative tests)?
Received on Tuesday, 23 May 2017 13:23:41 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:13 UTC