Re: Writing tests where browsers are known to not be conforming

On Thu, Jun 12, 2014 at 9:17 AM, James Graham <james@hoppipolla.co.uk>
wrote:

> On 12/06/14 17:00, Dirk Pranke wrote:
> > On Thu, Jun 12, 2014 at 7:56 AM, James Graham <james@hoppipolla.co.uk>
> > wrote:
> >
> >> On 12/06/14 15:31, Patrik Höglund wrote:
> >>> Hi!
> >>>
> >>> Posting here by request of dom@w3.org.
> >>>
> >>> I'm writing some testharness.js-based conformance tests for the
> >> getUserMedia
> >>> spec <http://dev.w3.org/2011/webrtc/editor/getusermedia.html>. I was
> >>> planning to check those in here
> >>> <https://github.com/w3c/web-platform-tests/tree/master/webrtc>. We
> have
> >> a
> >>> mechanism for chromium/blink which can run these tests continuously so
> we
> >>> know we don't regress. However, since the getUserMedia spec is quite
> new
> >>> and evolving, Chrome and Firefox fail a bunch of the test cases (e.g.
> >> that
> >>> attributes aren't in the right place, methods aren't implemented yet,
> >> etc).
> >>>
> >>> Since we want the tests running continuously to not fail all the time,
> is
> >>> there some established way of "disabling" these tests in continuous
> >>> integration? Like, could we pass a parameter
> ?dont_run_known_failing=true
> >>> where we keep a list of known broken test cases in the test file for
> each
> >>> browser?
> >>
> >> I don't know how blink are planning to integrate web-platform-tests in
> >> their CI.
> >
> >
> > This is actually documented at
> > http://www.chromium.org/blink/importing-the-w3c-tests , if anyone is
> > curious.
> >
> > As part of the import process, we maintain a blacklist of things not to
> > import (checked in to Blink), so skipping broken tests is already
> supported.
>
> So I didn't understand everything there but it wasn't clear if you
> support expectations at a lower level than per-file or not.
>
>
The Blink implementation of testharnessreport converts the assertions to a
series of text lines, e.g.:

PASS test 1
FAIL test 2
PASS test 3

It is possible to check in an expected version of the output for the test,
and that allows us to track tests where only some of the sub-tests pass. If
the expected output is not checked in, we assume that all of the sub-tests
are expected to pass.

I did notice that some tests were listed as being "broken" in rather
> trivial ways. It would be nice to see patches to fix those tests.


Yes.

-- Dirk