Re: web-platform-tests now running in Gecko CI system

On 09/09/14 13:00, Andreas Tolfsen wrote:
> ↪ James Graham <james@hoppipolla.co.uk>:
>> As of Friday the full [1] set of web-platform-tests are now running in
>> the Mozilla Continuous Integration (CI) system [2]. This means that they
>> are running on every single commit to the main Gecko development tree,
>> and that developers are responsible for keeping the tests from regressing.
> 
> This is a very big milestone, thank you for the hard work you've done!
> 
>> [3] Linux, Windows XP, Windows 7 and Windows 8 could probably be enabled
>> immediately, OSX seems to be more troublesome, and the harness currently
>> doesn't support running on Android.
> 
> Can you shed some light on what the apparent problems are with Mac
> OS?

It seems to be more unstable on OSX for some reason, with random tests
timing out with frequency that would be problematic in CI. I *think*
this is either a gecko problem or a marionette problem (the one time I
managed to reproduce it, it looked like something was broken in the
cache because all loads failed until I cleared the cache). It might be
possible to work around this by restarting the browser after every
directory of tests, or by allowing tests to be retried if they get an
unexpected result.

> Are there also other open issues which needs addressing?

Well, there is certainly more work to be done. In no particular order:

* SSL (i.e. https) support

* Alignment of semantics with CSS tests (multiple and optional matches)

* Fuzzy matching for (some) reftests

* WebDriver spec-test support

* Automated "manual" tests (e.g. via webdriver)

* Better reporting (e.g. construction of JSON report)

* Support for more platforms and browsers.

This is in addition to work on other components e.g. testharness.js
changes to improve testing with state-of-the-art APIs.

Some of the above are "easy" in that they just require time; others are
"hard" in the sense that it isn't really clear how to fix them in an
appropriate cross-browser way yet.

Received on Tuesday, 9 September 2014 12:32:59 UTC