Re: Writing tests where browsers are known to not be conforming

On 12/06/14 17:00, Dirk Pranke wrote:
> On Thu, Jun 12, 2014 at 7:56 AM, James Graham <james@hoppipolla.co.uk>
> wrote:
> 
>> On 12/06/14 15:31, Patrik Höglund wrote:
>>> Hi!
>>>
>>> Posting here by request of dom@w3.org.
>>>
>>> I'm writing some testharness.js-based conformance tests for the
>> getUserMedia
>>> spec <http://dev.w3.org/2011/webrtc/editor/getusermedia.html>. I was
>>> planning to check those in here
>>> <https://github.com/w3c/web-platform-tests/tree/master/webrtc>. We have
>> a
>>> mechanism for chromium/blink which can run these tests continuously so we
>>> know we don't regress. However, since the getUserMedia spec is quite new
>>> and evolving, Chrome and Firefox fail a bunch of the test cases (e.g.
>> that
>>> attributes aren't in the right place, methods aren't implemented yet,
>> etc).
>>>
>>> Since we want the tests running continuously to not fail all the time, is
>>> there some established way of "disabling" these tests in continuous
>>> integration? Like, could we pass a parameter ?dont_run_known_failing=true
>>> where we keep a list of known broken test cases in the test file for each
>>> browser?
>>
>> I don't know how blink are planning to integrate web-platform-tests in
>> their CI.
> 
> 
> This is actually documented at
> http://www.chromium.org/blink/importing-the-w3c-tests , if anyone is
> curious.
> 
> As part of the import process, we maintain a blacklist of things not to
> import (checked in to Blink), so skipping broken tests is already supported.

So I didn't understand everything there but it wasn't clear if you
support expectations at a lower level than per-file or not.

I did notice that some tests were listed as being "broken" in rather
trivial ways. It would be nice to see patches to fix those tests.

> I thought Patrik was asking whether we should have a similar list actually
> checked into web-platform-tests, but perhaps I misunderstood him.

Oh, if that's the question then the answer is pretty clearly "no" I
think. It doesn't make too sense to check in browser-specific
expectation files, much less import blacklists.

Received on Thursday, 12 June 2014 16:17:46 UTC