W3C home > Mailing lists > Public > public-test-infra@w3.org > October to December 2016

Stability testing of PRs

From: James Graham <james@hoppipolla.co.uk>
Date: Mon, 17 Oct 2016 15:42:50 +0100
To: "public-test-infra@w3.org" <public-test-infra@w3.org>
Cc: Rick Byers <rbyers@google.com>, Philip Jägenstedt <foolip@google.com>
Message-ID: <f089917a-8167-065c-e4f0-29a3c2787d90@hoppipolla.co.uk>
Unstable tests are one of the biggest problems with running 
web-platform-tests — or any tests — on browser CI infrastructure, and in 
particular I have found that unreliable web-platform-tests are one of 
the biggest problems when importing tests into the Mozilla CI system, 
often forcing me to perform multiple, slow, end-to-end runs, and disable 
tests, before landing the change.

In order to alleviate this problem, I implemented a travis job that 
checks submitted tests produce stable results in 10 runs of the latest 
public version of Firefox / Chrome. The code is in PR/3975 [1]. This PR 
is pending a release of the Firefox remote control library (marionette) 
and code review, but after those conditions are met I intend to turn it 
on as soon as possible. I would also like to add Edge and Safari; Edge 
seems possible using Appveyor, Safari may be possible on Travis. However 
given the relative difficulty of testing those browsers locally I don't 
intend to work on this immediately.

I expect there will be some cases where this job fails due to legitimate 
browser bugs causing instability. In that case I think that a comment 
indicating that the test author has investigated the issue and concluded 
that it must be a browser bug should be enough for an admin to merge in 
this case.

Does anyone have any concerns about adding this check?

[1] https://github.com/w3c/web-platform-tests/pull/3975
Received on Monday, 17 October 2016 14:43:18 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:12 UTC