Re: Recent instability in WPT

Thanks for the summary of the situation Mike, that's really helpful.

Most importantly, thanks to you and Bob for continuing to invest heavily in
the tooling and making using WPT easier and more enjoyable!  Keep up the
great work!

On Thu, Apr 6, 2017 at 3:39 PM, Mike Pennisi <mike@bocoup.com> wrote:

> Hello all,
>
> My name is Mike, and I've been contributing to the Web Platform Tests'
> infrastructure for the past few months. If you've submitted a pull request
> to
> the project in the past week, you may have experienced confusing results
> reported by the project's continuous integration system. Here's an
> overview of
> the different known issues and what we are doing to resolve them. Also
> included
> is a link to a dedicated ticket on GitHub.com's issue tracker; you can use
> these to check on the status of each bug.
>
> **Running unmodified tests.** Sometimes tests that are unrelated to a pull
> request are run by the "check stability" script. This can lead to spurious
> build failures when the extraneous tests are themselves found to be
> unstable.
> We're updating the inference logic to use more well-defined aspects of the
> TravisCI environment.
> https://github.com/w3c/web-platform-tests/issues/5408
>
> **Pull request fails with git error, "fatal: Invalid revision range"** Some
> reports contain a git error and have no actual test results. As above,
> we're
> updating the inference logic to use more well-defined aspects of the
> TravisCI
> environment.
> https://github.com/w3c/web-platform-tests/issues/5409
>
> **`master` branch build fails with git error, "fatal: ambiguous argument
> '^'**
> This only concerns so-called "push builds," against `master`, so
> (depending on
> your notification settings) you may receive an e-mail from TravisCI when it
> occurs. Likewise, we do not expect it to interfere with validation of pull
> requests. Please let us know if you are seeing it in other contexts. We've
> identified a bug in our internal git branch point" detection logic and
> will be
> correcting it shortly.
> https://github.com/w3c/web-platform-tests/issues/5410
>
> **Chrome runs zero tests.** The Chrome job occasionally reports "no tests
> run."
> This is because the ChromeDriver binary is failing to connect, cause still
> unknown. We're experimenting locally and reaching out to folks on the
> Chromium
> team for input.
> https://github.com/w3c/web-platform-tests/issues/5407
>
> **CSS tests reported as unstable in Chrome.** CSS tests that depend on
> loading
> external resources may fail sporadically if those resources are not ready
> when
> the screenshot is captured. The issue has information about best practices
> here, though not all of them are technically possible with the current
> infrastructure. We're considering infrastructure improvements that would
> delay
> the screenshot until the appropriate moment.
> https://github.com/w3c/web-platform-tests/issues/5412
>
> Finally, you may have seen references to a race condition in Chromium's
> rendering logic, as being discussed here:
> https://github.com/jugglinmike/chrome-screenshot-race/issues/1 . That has
> only
> been verified to affect tests that include image assets, and a temporary
> fix
> has been implemented in WPT. It is unlikely to be the source of further
> instability, so please double-check the above list before commenting there.
>
> Keep an eye on those open issues and bear with us as we resolve each in
> turn.
> If you experience some failure that isn't accounted for above, please
> reach out
> to either James Graham (jgraham on IRC and GitHub), Bob Holt (bobholt on
> IRC
> and GitHub) or me (jugglinmike on IRC and GitHub). Also, pass this message
> along to other contributors.
>
> Thanks,
> Mike
>
>