Re: WPT Developer Survey - June 2018

"Non-fatal assertions" allow authors to make multiple independent
assertions within a single test body. As James wrote, testharness.js
wasn't designed to support this kind of thing. We might be able to hack
it in via dynamically generated tests, something like:

    function nonfatal_assert(value, msg) {
      test(msg, function() { assert(value); });
    });

...but this would confuse the meaning of "sub-test" and likely
contribute to inconsistencies in total number of sub-tests across
browsers.

Even if there is a more reasonable implementation than my strawman, I
think the functionality is contrary to many of our goals. Implementing
it in any fashion will give contributors another way to "say" the same
thing. We're already struggling to document the current API. We'll
hopefully get over that soon, but this would still tend to make WPT more
irregular and less approachable.

The other survey feedback about simplifying tests concern the
requirement to express asynchronous assertions within callbacks to the
`step` method.  This is necessary so the harness can associate failures
with sub-tests.  In general, uncaught exceptions (as produced by
assertions that aren't "wrapped") can't be associated because sub-tests
declared with the `async_test` function are run in parallel.

If sub-tests were always executed in series [1], then uncaught
exceptions could be reliably associated with sub-tests, and contributors
wouldn't need to use `step`.

It's unlikely we'd be able to change the behavior for existing
`async_test`s [2]. We might be able to satisfy contributors by offering
an opt-in for new tests. That would also be more API to document and
learn, though.

We might take advantage of the fact that `promise_test`s are *already*
executed in series. testharness.js could react to uncaught exceptions by
first checking if there were no `async_tests` being executed and in that
case, trigger failure in the active `promise_test`.  Then our advice to
contributors looking to be more concise would be simply, "use
`promise_test`."

I ran an experiment to see how many tests that use `promise_test` have
uncaught exceptions in Firefox and Chrome [3]. There are 49 such tests.
That makes changing the semantics in this way seem tractable. Do you
folks think it's a good idea?

[1] https://github.com/web-platform-tests/wpt/issues/6002
[2]
https://github.com/web-platform-tests/wpt/issues/5970#issuecomment-302810259
[3] Commit:
https://github.com/bocoup/wpt/commit/38972d9d995a652dbe4cbe13320bc7a9b6d4c617
    Consolidated results:
https://gist.github.com/jugglinmike/a67acb16a6a877628abe1684daeede04


On 09/28/2018 01:30 PM, Philip Jägenstedt wrote:
> On Fri, Sep 28, 2018, 19:10 Mike Pennisi <mike@bocoup.com
> <mailto:mike@bocoup.com>> wrote:
>
>     Thanks, Simon! Some interesting comments have been made on the
>     "Google Ecosys Infra" mailing list:
>
>     https://groups.google.com/a/chromium.org/forum/m/#%21topic/ecosystem-infra/HuqN39mcBNk
>
>     I assume this list has a wider readership, so if possible, could
>     we keep the conversation focused here?
>
> Sure :)
>
> [repeat question about test harness.js ergonomics]