Recent instability in WPT

Hello all,

My name is Mike, and I've been contributing to the Web Platform Tests'
infrastructure for the past few months. If you've submitted a pull 
request to
the project in the past week, you may have experienced confusing results
reported by the project's continuous integration system. Here's an 
overview of
the different known issues and what we are doing to resolve them. Also 
included
is a link to a dedicated ticket on GitHub.com's issue tracker; you can use
these to check on the status of each bug.

**Running unmodified tests.** Sometimes tests that are unrelated to a pull
request are run by the "check stability" script. This can lead to spurious
build failures when the extraneous tests are themselves found to be 
unstable.
We're updating the inference logic to use more well-defined aspects of the
TravisCI environment.
https://github.com/w3c/web-platform-tests/issues/5408

**Pull request fails with git error, "fatal: Invalid revision range"** Some
reports contain a git error and have no actual test results. As above, we're
updating the inference logic to use more well-defined aspects of the 
TravisCI
environment.
https://github.com/w3c/web-platform-tests/issues/5409

**`master` branch build fails with git error, "fatal: ambiguous argument 
'^'**
This only concerns so-called "push builds," against `master`, so 
(depending on
your notification settings) you may receive an e-mail from TravisCI when it
occurs. Likewise, we do not expect it to interfere with validation of pull
requests. Please let us know if you are seeing it in other contexts. We've
identified a bug in our internal git branch point" detection logic and 
will be
correcting it shortly.
https://github.com/w3c/web-platform-tests/issues/5410

**Chrome runs zero tests.** The Chrome job occasionally reports "no 
tests run."
This is because the ChromeDriver binary is failing to connect, cause still
unknown. We're experimenting locally and reaching out to folks on the 
Chromium
team for input.
https://github.com/w3c/web-platform-tests/issues/5407

**CSS tests reported as unstable in Chrome.** CSS tests that depend on 
loading
external resources may fail sporadically if those resources are not 
ready when
the screenshot is captured. The issue has information about best practices
here, though not all of them are technically possible with the current
infrastructure. We're considering infrastructure improvements that would 
delay
the screenshot until the appropriate moment.
https://github.com/w3c/web-platform-tests/issues/5412

Finally, you may have seen references to a race condition in Chromium's
rendering logic, as being discussed here:
https://github.com/jugglinmike/chrome-screenshot-race/issues/1 . That 
has only
been verified to affect tests that include image assets, and a temporary fix
has been implemented in WPT. It is unlikely to be the source of further
instability, so please double-check the above list before commenting there.

Keep an eye on those open issues and bear with us as we resolve each in 
turn.
If you experience some failure that isn't accounted for above, please 
reach out
to either James Graham (jgraham on IRC and GitHub), Bob Holt (bobholt on IRC
and GitHub) or me (jugglinmike on IRC and GitHub). Also, pass this message
along to other contributors.

Thanks,
Mike

Received on Thursday, 6 April 2017 19:40:35 UTC