Helping implementations test error messages

Hi testing folks,

In implementing the Web Bluetooth spec, we'd like to test that it produces
good error messages for developers. However, we'd also like our tests to be
usable by other implementations without effectively making the error
messages part of the specification. testharness.js doesn't currently have
support for this, so I'm looking for guidance on what to add to enable it.

An initial proposal from the blink-dev thread on this (
https://groups.google.com/a/chromium.org/d/topic/blink-dev/8YfKmyq7ugQ/discussion)
is to have assert_throws(func, {name: 'FooError', printMessage: true}) call
printErrorMessage(e.message), which would be provided by
testharnessreport.js. Then Blink would put its expected error messages in
our -expected.txt files, and other implementations would be free to check
their error messages or not, as they wish. This has the downsides that
every implementation has to check the same set of error messages, if they
check any, and that it can't really be used from concurrent tests.

We are, of course, open to better ideas from this group.

Jeffrey

Received on Thursday, 23 July 2015 12:18:39 UTC