- From: Mike Pennisi <mike@bocoup.com>
- Date: Fri, 28 Sep 2018 16:03:03 -0400
- To: Philip Jägenstedt <foolip@google.com>
- Cc: public-test-infra <public-test-infra@w3.org>
- Message-ID: <339b15f7-2c68-cdf1-0c30-d56476d87c34@bocoup.com>
"Non-fatal assertions" allow authors to make multiple independent assertions within a single test body. As James wrote, testharness.js wasn't designed to support this kind of thing. We might be able to hack it in via dynamically generated tests, something like: function nonfatal_assert(value, msg) { test(msg, function() { assert(value); }); }); ...but this would confuse the meaning of "sub-test" and likely contribute to inconsistencies in total number of sub-tests across browsers. Even if there is a more reasonable implementation than my strawman, I think the functionality is contrary to many of our goals. Implementing it in any fashion will give contributors another way to "say" the same thing. We're already struggling to document the current API. We'll hopefully get over that soon, but this would still tend to make WPT more irregular and less approachable. The other survey feedback about simplifying tests concern the requirement to express asynchronous assertions within callbacks to the `step` method. This is necessary so the harness can associate failures with sub-tests. In general, uncaught exceptions (as produced by assertions that aren't "wrapped") can't be associated because sub-tests declared with the `async_test` function are run in parallel. If sub-tests were always executed in series [1], then uncaught exceptions could be reliably associated with sub-tests, and contributors wouldn't need to use `step`. It's unlikely we'd be able to change the behavior for existing `async_test`s [2]. We might be able to satisfy contributors by offering an opt-in for new tests. That would also be more API to document and learn, though. We might take advantage of the fact that `promise_test`s are *already* executed in series. testharness.js could react to uncaught exceptions by first checking if there were no `async_tests` being executed and in that case, trigger failure in the active `promise_test`. Then our advice to contributors looking to be more concise would be simply, "use `promise_test`." I ran an experiment to see how many tests that use `promise_test` have uncaught exceptions in Firefox and Chrome [3]. There are 49 such tests. That makes changing the semantics in this way seem tractable. Do you folks think it's a good idea? [1] https://github.com/web-platform-tests/wpt/issues/6002 [2] https://github.com/web-platform-tests/wpt/issues/5970#issuecomment-302810259 [3] Commit: https://github.com/bocoup/wpt/commit/38972d9d995a652dbe4cbe13320bc7a9b6d4c617 Consolidated results: https://gist.github.com/jugglinmike/a67acb16a6a877628abe1684daeede04 On 09/28/2018 01:30 PM, Philip Jägenstedt wrote: > On Fri, Sep 28, 2018, 19:10 Mike Pennisi <mike@bocoup.com > <mailto:mike@bocoup.com>> wrote: > > Thanks, Simon! Some interesting comments have been made on the > "Google Ecosys Infra" mailing list: > > https://groups.google.com/a/chromium.org/forum/m/#%21topic/ecosystem-infra/HuqN39mcBNk > > I assume this list has a wider readership, so if possible, could > we keep the conversation focused here? > > Sure :) > > [repeat question about test harness.js ergonomics]
Received on Friday, 28 September 2018 20:03:41 UTC