W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2017

Tentative tests.

From: Mike West <mkwst@google.com>
Date: Tue, 23 May 2017 15:10:28 +0200
Message-ID: <CAKXHy=epPz0BJE1BwSaX3nvKsGnfwSdWOTskaBbMAGpSF70o=A@mail.gmail.com>
To: public-test-infra@w3.org
Cc: Patrick Kettner <patket@microsoft.com>, Philip J├Ągenstedt <foolip@chromium.org>, Mark Dittmer <markdittmer@chromium.org>, Ojan Vafai <ojan@chromium.org>, Boris Zbarsky <bzbarsky@mit.edu>, Anne van Kesteren <annevk@annevk.nl>
Following up on "when automerging of tests goes awry
<https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg25>"
and "Dealing with prove-it-then-spec-it situations in web-platform-tests
<https://groups.google.com/a/chromium.org/d/msg/blink-dev/LDc3RgrrldU/a1Gn0ld9EQAJ>"
(and CCing various folks who were involved in those conversations). Based
on those conversations, it sounds like there's something approaching
consensus that it would be good to support "tentative" tests in WPT that
run ahead of the relevant specification. This allows us to gather
implementation experience to inform our decisions about the "right" answer
to spec questions, and allows us to share that experience easily with each
other.

In order to ensure that we're on the same page about the status of these
tests, I'd propose that when uploading tentative tests, developers choose
one of the following identification mechanisms:

1.  Put the tests in a new file with a `-tentative.*` suffix (e.g.
`//content-security-policy/script-src/new-test-tentative.sub.html`).

2.  Put the tests in a `tentative` subdirectory of an existing suite (e.g.
`//content-security-policy/tentative/new-script-src-test.sub.html`).

These both seem fairly unambiguous, and I see good use-cases for each.

WDYT?

-mike
Received on Tuesday, 23 May 2017 13:11:22 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:13 UTC