Re: WPT: PASS despite failed assertion

On 20/10/16 22:13, Geoffrey Sneddon wrote:
> On Sat, Oct 15, 2016 at 4:30 PM, Jack Bates <z1a8te@nottheoilrig.com> wrote:
>> I noticed a couple of tests that pass even if one of their assertions fails.
>> To see what I mean, add an assert_true(false) to
>> fetch/api/cors/cors-preflight-redirect.js, for one.
>> My browser logs the AssertionError, but testharness.js doesn't catch it, so
>> the test passes despite the failed assertion.
>> Is this a known issue? Can testharness.js be improved so the failed
>> assertion doesn't just vanish?
>> I think the issue can be addressed superficially in the tests themselves,
>> but at a higher level, is there something testharness.js can do to avoid
>> concealing the issue by swallowing the failed assertion?
>
> Where in fetch/api/cors/cors-preflight-redirect.js? Asserts are only
> picked up within tests (with some magic for single-file tests where
> the whole file is considered tests), hence only asserts within the
> anonymous function provided as the first argument to promise_test will
> get picked up. (Hopefully that should always get picked up; I don't
> know well how our support for promises works.)
>
> One approach could be to have an error event listener on the document
> that sets the whole test (i.e., all subtests) into some error state if
> there were an (uncaught) assert reach it. Anyone else have any
> opinion?

There is an error listener that sets the overall file status to error if 
there is an uncaught exception. I think this is reasonable (and any 
other change would be backwards-incompatible), since an exception is 
either a bug in the test or a fundamental problem with the browser under 
test (e.g. it doesn't support the APIs being tested).

Received on Thursday, 20 October 2016 21:33:27 UTC