W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2014

Re: Unstable tests

From: James Graham <james@hoppipolla.co.uk>
Date: Fri, 18 Apr 2014 23:51:00 +0100
Message-ID: <5351AC54.40903@hoppipolla.co.uk>
To: "public-test-infra@w3.org" <public-test-infra@w3.org>
On 25/03/14 11:15, James Graham wrote:
> Using data from a set of web-platform-tests runs in desktop Firefox, I
> have a list of tests that aren't producing consistent results. This data
> is based on 10 runs per platform for 5 different platforms (2 Linux, 3
> OS X). Instability is considered per-platform (so a test that
> consistently produces one result on OSX and another on Linux is
> considered stable). The list of tests that produced unstable results in
> any configuration is at [1]. For the purposes of presentation I squashed
> the results down into a single list without platform information, but I
> can change that if needed.

With some work on test stability, I now have a new list [1]. This is 
from 45 runs across 4 different platforms, so it won't find tests that 
are only rarely unstable (roughly speaking this should be about 90% of 
tests that change results at least one time in 20).

Help fixing the remaining tests, or identifying gecko bugs, is welcome, 
as always.

[1] http://hoppipolla.co.uk/410/unstable-1.txt
Received on Friday, 18 April 2014 22:51:24 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:10 UTC