W3C home > Mailing lists > Public > public-test-infra@w3.org > January to March 2014

Unstable tests

From: James Graham <james@hoppipolla.co.uk>
Date: Tue, 25 Mar 2014 11:15:51 +0000
Message-ID: <53316567.2060206@hoppipolla.co.uk>
To: "public-test-infra@w3.org" <public-test-infra@w3.org>
Using data from a set of web-platform-tests runs in desktop Firefox, I 
have a list of tests that aren't producing consistent results. This data 
is based on 10 runs per platform for 5 different platforms (2 Linux, 3 
OS X). Instability is considered per-platform (so a test that 
consistently produces one result on OSX and another on Linux is 
considered stable). The list of tests that produced unstable results in 
any configuration is at [1]. For the purposes of presentation I squashed 
the results down into a single list without platform information, but I 
can change that if needed.

This set of tests represents about 2% of the top level test files in the 
repository. In order to use the testsuite in the Mozilla CI system (or 
in any other CI system), the rate of instability has to be much lower 
than that. Therefore it is necessary to determine why these tests are 
not giving consistent results and take appropriate action.

In the best case the problems will be largely with the tests themselves, 
either doing something non-deterministic or just having too short a 
timeout, or whatever. In this case we need to fix the test. In some 
cases the instability may be due to non-determinism in Firefox, in which 
case I may (unfortunately) have to disable the test locally until the 
underlying issue is fixed.

If you have any time to help investigate the issues with these tests, 
particularly for tests that you own (i.e. ones that you wrote), it would 
be much appreciated.

[1] http://hoppipolla.co.uk/410/unstable.txt
Received on Tuesday, 25 March 2014 11:16:15 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:20:34 UTC