[testing] "unstable" tests: eventsource, workers, websockets, webmessaging, webstorage

James identified tests "that aren't producing consistent results" 
<http://hoppipolla.co.uk/410/unstable.txt>. For WebApps, the test suites 
indentified are: eventsource, workers, websockets, webmessaging and 
webstorage. Please review these tests and if needed, create an issue 
and/or submit a fix.

-Thanks, AB

-------- Original Message --------
Subject: 	Unstable tests
Resent-Date: 	Tue, 25 Mar 2014 11:16:16 +0000
Resent-From: 	<public-test-infra@w3.org>
Date: 	Tue, 25 Mar 2014 11:15:51 +0000
From: 	ext James Graham <james@hoppipolla.co.uk>
To: 	public-test-infra@w3.org <public-test-infra@w3.org>



Using data from a set of web-platform-tests runs in desktop Firefox, I
have a list of tests that aren't producing consistent results. This data
is based on 10 runs per platform for 5 different platforms (2 Linux, 3
OS X). Instability is considered per-platform (so a test that
consistently produces one result on OSX and another on Linux is
considered stable). The list of tests that produced unstable results in
any configuration is at [1]. For the purposes of presentation I squashed
the results down into a single list without platform information, but I
can change that if needed.

This set of tests represents about 2% of the top level test files in the
repository. In order to use the testsuite in the Mozilla CI system (or
in any other CI system), the rate of instability has to be much lower
than that. Therefore it is necessary to determine why these tests are
not giving consistent results and take appropriate action.

In the best case the problems will be largely with the tests themselves,
either doing something non-deterministic or just having too short a
timeout, or whatever. In this case we need to fix the test. In some
cases the instability may be due to non-determinism in Firefox, in which
case I may (unfortunately) have to disable the test locally until the
underlying issue is fixed.

If you have any time to help investigate the issues with these tests,
particularly for tests that you own (i.e. ones that you wrote), it would
be much appreciated.

[1] http://hoppipolla.co.uk/410/unstable.txt

Received on Monday, 7 April 2014 13:31:26 UTC