Re: Unstable tests

This is extremely useful. Thanks for doing it.

Do you think there is value in opening issues for each of these (or
groups of these) on Github? That's relatively easy to do with GitHub's
API. I'd be happy to look into it if you think there's value there.

Sent from my mobile phone.

On Mar 25, 2014, at 12:15, James Graham <james@hoppipolla.co.uk> wrote:

> Using data from a set of web-platform-tests runs in desktop Firefox, I have a list of tests that aren't producing consistent results. This data is based on 10 runs per platform for 5 different platforms (2 Linux, 3 OS X). Instability is considered per-platform (so a test that consistently produces one result on OSX and another on Linux is considered stable). The list of tests that produced unstable results in any configuration is at [1]. For the purposes of presentation I squashed the results down into a single list without platform information, but I can change that if needed.
>
> This set of tests represents about 2% of the top level test files in the repository. In order to use the testsuite in the Mozilla CI system (or in any other CI system), the rate of instability has to be much lower than that. Therefore it is necessary to determine why these tests are not giving consistent results and take appropriate action.
>
> In the best case the problems will be largely with the tests themselves, either doing something non-deterministic or just having too short a timeout, or whatever. In this case we need to fix the test. In some cases the instability may be due to non-determinism in Firefox, in which case I may (unfortunately) have to disable the test locally until the underlying issue is fixed.
>
> If you have any time to help investigate the issues with these tests, particularly for tests that you own (i.e. ones that you wrote), it would be much appreciated.
>
> [1] http://hoppipolla.co.uk/410/unstable.txt
>

Received on Tuesday, 25 March 2014 14:33:32 UTC