Writing tests where browsers are known to not be conforming

Hi!

Posting here by request of dom@w3.org.

I'm writing some testharness.js-based conformance tests for the getUserMedia
spec <http://dev.w3.org/2011/webrtc/editor/getusermedia.html>. I was
planning to check those in here
<https://github.com/w3c/web-platform-tests/tree/master/webrtc>. We have a
mechanism for chromium/blink which can run these tests continuously so we
know we don't regress. However, since the getUserMedia spec is quite new
and evolving, Chrome and Firefox fail a bunch of the test cases (e.g. that
attributes aren't in the right place, methods aren't implemented yet, etc).

Since we want the tests running continuously to not fail all the time, is
there some established way of "disabling" these tests in continuous
integration? Like, could we pass a parameter ?dont_run_known_failing=true
where we keep a list of known broken test cases in the test file for each
browser?

BR,
/ P