Re: WPT: PASS despite failed assertion

On Sat, Oct 15, 2016 at 4:30 PM, Jack Bates <z1a8te@nottheoilrig.com> wrote:
> I noticed a couple of tests that pass even if one of their assertions fails.
> To see what I mean, add an assert_true(false) to
> fetch/api/cors/cors-preflight-redirect.js, for one.
> My browser logs the AssertionError, but testharness.js doesn't catch it, so
> the test passes despite the failed assertion.
> Is this a known issue? Can testharness.js be improved so the failed
> assertion doesn't just vanish?
> I think the issue can be addressed superficially in the tests themselves,
> but at a higher level, is there something testharness.js can do to avoid
> concealing the issue by swallowing the failed assertion?

Where in fetch/api/cors/cors-preflight-redirect.js? Asserts are only
picked up within tests (with some magic for single-file tests where
the whole file is considered tests), hence only asserts within the
anonymous function provided as the first argument to promise_test will
get picked up. (Hopefully that should always get picked up; I don't
know well how our support for promises works.)

One approach could be to have an error event listener on the document
that sets the whole test (i.e., all subtests) into some error state if
there were an (uncaught) assert reach it. Anyone else have any
opinion?

/g

Received on Thursday, 20 October 2016 21:14:01 UTC