W3C home > Mailing lists > Public > public-audio@w3.org > April to June 2015

Re: Calling All Tests

From: Karl Tomlinson <karlt+public-audio@karlt.net>
Date: Tue, 09 Jun 2015 13:42:30 +1200
To: Joe Berkovitz <joe@noteflight.com>
Cc: Raymond Toy <rtoy@google.com>, Chris Lowis <chris.lowis@gmail.com>, Paul Adenot <padenot@mozilla.com>, Audio Working Group <public-audio@w3.org>
Message-ID: <87616xu06x.fsf@karlt.net>
Joe Berkovitz writes:

>> My assumption here was that, say, FF found a bug in their implementation
>> and want a test for that and pushes that to the repo. Which then causes
>> Chrome to fail that test (for whatever reason).  We wouldn't want to block
>> FF from having a regression test, but we wouldn't want to be blocked from
>> updating because the test will cause failures in our test bots.
> Perhaps we need a way to gate tests in the suite so that you are able to
> arbitrarily exclude ones that are not applicable for some reason (or
> arbitrarily include the ones you want).

Web platform tests are run on Gecko using wptrunner.
This provides a mechanism [1] to annotate the current result of
the test or to disable if the result is not deterministic.

[1] http://wptrunner.readthedocs.org/en/latest/expectation.html

Sometimes tests pulled into Gecko after development against
another browser identify bugs in Gecko, which can later be fixed,
and Gecko sometimes identifies bugs in the tests, but it is not
difficult to deal with these, and it is beneficial to see where
such differences exist.
Received on Tuesday, 9 June 2015 01:44:01 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 9 June 2015 01:44:01 UTC