W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2017

[update] Recent instability in WPT

From: Mike Pennisi <mike@bocoup.com>
Date: Tue, 25 Apr 2017 11:49:53 -0400
To: public-test-infra@w3.org
Message-ID: <aed36c99-61f7-8fdf-bb2e-ad87edcca4e8@bocoup.com>
Hi folks,

A few weeks ago, I sent a message to this list explaining various issues
that were causing intermittent failures in WPT's patch validation process
[1]. We've since made good progress in addressing those issues. I'm writing
now to let you know where we stand today.

**Running unmodified tests.** This was a result of relying on a
poorly-documented feature of the GitHub API. We've updated the "changed
file" inference logic to use a more direct approach and have not experienced
this problem since.

Original report: https://github.com/w3c/web-platform-tests/issues/5408
Resolution: https://github.com/w3c/web-platform-tests/pull/5416

**Pull request fails with git error, "fatal: Invalid revision range"** and
**`master` branch build fails with git error, "fatal: ambiguous argument
'^'** These problems were expressions of the same bug; the error message
happened to be contextual to the type of TravisCI build (i.e. "branch" or
"pull request").  In both cases, the error would occur when new commits were
pushed to a branch and the state of `master` had since advanced beyond its
original commit. Here too, an extension to the "changed files" inference
logic has resolved the issue.

Original report: https://github.com/w3c/web-platform-tests/issues/5409
Resolution: https://github.com/w3c/web-platform-tests/pull/5411

**Chrome runs zero tests.** This is caused by a long-standing issue with
"headless" Linux platforms. That we began seeing it with increased frequency
earlier this month may be due to the concurrent release of a new version of
ChromeDriver. In any case, we have implemented a known-good workaround for
WPT.  More importantly, we have pushed on solutions father "upstream", both
in the Chromium project and in the underlying systems infrastructure, Glib.

Original report: https://github.com/w3c/web-platform-tests/issues/5407
Resolution:
- https://github.com/w3c/web-platform-tests/pull/5626
- (pending) https://bugs.chromium.org/p/chromium/issues/detail?id=713947
- (pending) https://bugzilla.gnome.org/show_bug.cgi?id=738620

**CSS tests reported as unstable in Chrome.** We are still investigating the
exact conditions that cause this failure. Fortunately, it is more rare than
we originally estimated. But until the underlying issue is resolved, we will
keep the issue open and up-to-date.

Original report: https://github.com/w3c/web-platform-tests/issues/5412
Resolution: (pending)

Also note that the recent merging of the project's various submodules [2]
will allow us to more thoroughly test the infrastructure [3] and avoid many
of these issues moving forward.

Many thanks to bobholt, gsnedders, and jgraham, all of whom worked under
pressure to help contributors understand the issues, resolve the underlying
problem, and generally keep the project moving forward through the stormy
weather.
Received on Tuesday, 25 April 2017 15:50:28 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 25 April 2017 15:50:29 UTC