Re: WPT Developer Survey - June 2018

Phillip reminded me that we've already discussed this, and that further,
I myself argued *against* it [1]. Basically, `step` is already avoidable
if authors are willing to use `promise_test` and define promises
rigorously. That might not be a satisfying answer for some folks,
though, since "defining promises rigorously" also entails more ceremony
(e.g. the more involved parts of [2]).

Still, what I like about that solution is it preserves the meaning of a
test error. If we interpret all uncaught exceptions as failures, we
won't be able to detect truly exceptional circumstances--test bugs will
be reported as though they were conformance violations. There's value in
explicitness (whether via `step` or picky promises), even if authors
find it cumbersome. Maybe they'd be more accepting if we communicated
that value more clearly.

[1] https://github.com/web-platform-tests/wpt/issues/11347
[2] https://github.com/web-platform-tests/wpt/pull/13165


On 09/28/2018 04:03 PM, Mike Pennisi wrote:
>
> "Non-fatal assertions" allow authors to make multiple independent
> assertions within a single test body. As James wrote, testharness.js
> wasn't designed to support this kind of thing. We might be able to hack
> it in via dynamically generated tests, something like:
>
>     function nonfatal_assert(value, msg) {
>       test(msg, function() { assert(value); });
>     });
>
> ...but this would confuse the meaning of "sub-test" and likely
> contribute to inconsistencies in total number of sub-tests across
> browsers.
>
> Even if there is a more reasonable implementation than my strawman, I
> think the functionality is contrary to many of our goals. Implementing
> it in any fashion will give contributors another way to "say" the same
> thing. We're already struggling to document the current API. We'll
> hopefully get over that soon, but this would still tend to make WPT more
> irregular and less approachable.
>
> The other survey feedback about simplifying tests concern the
> requirement to express asynchronous assertions within callbacks to the
> `step` method.  This is necessary so the harness can associate failures
> with sub-tests.  In general, uncaught exceptions (as produced by
> assertions that aren't "wrapped") can't be associated because sub-tests
> declared with the `async_test` function are run in parallel.
>
> If sub-tests were always executed in series [1], then uncaught
> exceptions could be reliably associated with sub-tests, and contributors
> wouldn't need to use `step`.
>
> It's unlikely we'd be able to change the behavior for existing
> `async_test`s [2]. We might be able to satisfy contributors by offering
> an opt-in for new tests. That would also be more API to document and
> learn, though.
>
> We might take advantage of the fact that `promise_test`s are *already*
> executed in series. testharness.js could react to uncaught exceptions by
> first checking if there were no `async_tests` being executed and in that
> case, trigger failure in the active `promise_test`.  Then our advice to
> contributors looking to be more concise would be simply, "use
> `promise_test`."
>
> I ran an experiment to see how many tests that use `promise_test` have
> uncaught exceptions in Firefox and Chrome [3]. There are 49 such tests.
> That makes changing the semantics in this way seem tractable. Do you
> folks think it's a good idea?
>
> [1] https://github.com/web-platform-tests/wpt/issues/6002
> [2]
> https://github.com/web-platform-tests/wpt/issues/5970#issuecomment-302810259
> [3] Commit:
> https://github.com/bocoup/wpt/commit/38972d9d995a652dbe4cbe13320bc7a9b6d4c617
>     Consolidated results:
> https://gist.github.com/jugglinmike/a67acb16a6a877628abe1684daeede04
>
>
> On 09/28/2018 01:30 PM, Philip Jägenstedt wrote:
>> On Fri, Sep 28, 2018, 19:10 Mike Pennisi <mike@bocoup.com
>> <mailto:mike@bocoup.com>> wrote:
>>
>>     Thanks, Simon! Some interesting comments have been made on the
>>     "Google Ecosys Infra" mailing list:
>>
>>     https://groups.google.com/a/chromium.org/forum/m/#%21topic/ecosystem-infra/HuqN39mcBNk
>>
>>     I assume this list has a wider readership, so if possible, could
>>     we keep the conversation focused here?
>>
>> Sure :)
>>
>> [repeat question about test harness.js ergonomics]
>