W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2014

Writing tests where browsers are known to not be conforming

From: Patrik Höglund <phoglund@google.com>
Date: Thu, 12 Jun 2014 16:31:56 +0200
Message-ID: <CAF+keRF2utQ6yMaSoHFjz-LqDRy5_Nm9MnUwZZR_9qHX=A0jpw@mail.gmail.com>
To: public-test-infra@w3.org

Posting here by request of dom@w3.org.

I'm writing some testharness.js-based conformance tests for the getUserMedia
spec <http://dev.w3.org/2011/webrtc/editor/getusermedia.html>. I was
planning to check those in here
<https://github.com/w3c/web-platform-tests/tree/master/webrtc>. We have a
mechanism for chromium/blink which can run these tests continuously so we
know we don't regress. However, since the getUserMedia spec is quite new
and evolving, Chrome and Firefox fail a bunch of the test cases (e.g. that
attributes aren't in the right place, methods aren't implemented yet, etc).

Since we want the tests running continuously to not fail all the time, is
there some established way of "disabling" these tests in continuous
integration? Like, could we pass a parameter ?dont_run_known_failing=true
where we keep a list of known broken test cases in the test file for each

/ P
Received on Thursday, 12 June 2014 14:36:58 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:10 UTC