W3C home > Mailing lists > Public > public-test-infra@w3.org > October to December 2016

Re: Stability testing of PRs

From: Philip Jägenstedt <foolip@google.com>
Date: Tue, 18 Oct 2016 12:03:46 +0000
Message-ID: <CAARdPYfBnSULCoros6D-B8X_EtCjBnN8RwoUk=neOrhTKKJkzA@mail.gmail.com>
To: James Graham <james@hoppipolla.co.uk>, "public-test-infra@w3.org" <public-test-infra@w3.org>
Cc: Rick Byers <rbyers@google.com>
That sounds fantastic, James!

If you've modified a common.js but no tests, will there be a way to
manually trigger a run for the tests you happen to know are affected?

Using the same infrastructure, would it be possible to paste the test
results into a comment in the PR after every change?

On Mon, Oct 17, 2016 at 4:42 PM James Graham <james@hoppipolla.co.uk> wrote:

> Unstable tests are one of the biggest problems with running
> web-platform-tests — or any tests — on browser CI infrastructure, and in
> particular I have found that unreliable web-platform-tests are one of
> the biggest problems when importing tests into the Mozilla CI system,
> often forcing me to perform multiple, slow, end-to-end runs, and disable
> tests, before landing the change.
> In order to alleviate this problem, I implemented a travis job that
> checks submitted tests produce stable results in 10 runs of the latest
> public version of Firefox / Chrome. The code is in PR/3975 [1]. This PR
> is pending a release of the Firefox remote control library (marionette)
> and code review, but after those conditions are met I intend to turn it
> on as soon as possible. I would also like to add Edge and Safari; Edge
> seems possible using Appveyor, Safari may be possible on Travis. However
> given the relative difficulty of testing those browsers locally I don't
> intend to work on this immediately.
> I expect there will be some cases where this job fails due to legitimate
> browser bugs causing instability. In that case I think that a comment
> indicating that the test author has investigated the issue and concluded
> that it must be a browser bug should be enough for an admin to merge in
> this case.
> Does anyone have any concerns about adding this check?
> [1] https://github.com/w3c/web-platform-tests/pull/3975
Received on Tuesday, 18 October 2016 12:04:30 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:12 UTC