W3C home > Mailing lists > Public > public-test-infra@w3.org > July to September 2016

Re: Data on flakiness of web platform tests

From: James Graham <james@hoppipolla.co.uk>
Date: Fri, 23 Sep 2016 18:05:47 +0100
To: public-test-infra@w3.org
Message-ID: <60d32603-84ab-7022-8691-9c2a8d41a6bb@hoppipolla.co.uk>
On 23/09/16 16:05, Rick Byers wrote:
> Geoffrey was asking for data on what web-platform-tests were flaky.  I
> don't have great data from blink (since we generally don't run the ones
> that are flaky) but here is what I do have:
>
>   * All flaky blink tests on Linux
>     <https://test-results.appspot.com/dashboards/flakiness_dashboard.html#showFlaky=true&builder=chromium.webkit%3AWebKit%20Linux>
>     - search this list for imported/wpt to find just the
>     web-platform-test ones
>   * List of web-platform-tests and csswg-tests that are blacklisted from
>     our import process
>     <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/W3CImportExpectations?type=cs&q=file:W3CImport&sq=package:chromium&l=1>
>     - could be for multiple reasons
>       o There are also some listed here
>         <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/TestExpectations> with
>         imported/wpt paths
>
> Other data?

https://dxr.mozilla.org/mozilla-central/search?q=path%3Atesting%2Fweb-platform%2Fmeta+disabled%3A&redirect=false

is the list of all the tests that we have disabled for some reason or 
another(that includes both top-level tests and subtests, although there 
are far more of the former than the latter). Usually there is a bug 
number you can dig into for more information.
Received on Friday, 23 September 2016 17:06:15 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:12 UTC