Re: Specifications with matching test APIs

On Fri, Apr 21, 2017 at 6:11 PM Rick Byers <rbyers@chromium.org> wrote:

> Thanks Reilly, I support continuing to move in this direction!
>
> Are you planning in shipping this API in release Chrome builds behind a
> flag?  I think it would be reasonable to modify the WPT infrastructure
> (stability_checker, dashboard) to pass a --enable-testing-apis flag
> (although we might need to consider the security implications of that for
> the WPT infrastructure running potentially untrusted test patches).  But I
> don't think we'd want to use content_shell (or even Chromium builds) in
> that infrastructure - at least not in place of Chrome builds.
>

The ability to override the Mojo services provided to the renderer from the
renderer itself (which is how Chromium's polyfill for this API is
implemented) is only available in content_shell when the --run-layout-test
parameter is passed. There have been discussions with the Mojo team about
making this available in production Chrome builds when a flag is enabled.
It would have to be a flag which displays the "unsupported flag, security
and stability will suffer" infobar because it effectively allows arbitrary
JavaScript to run with the privileges of the renderer.

What's your thoughts on writing tests which could be either manual tests
> ("attach a device with the following characteristics, etc..") or automated
> tests?
>

I think this is a good idea. I've seen other tests marked "-manual" which
can nonetheless be run in an automated fashion if the right hooks are in
place. That would be possible here but the necessary device firmware has
not been written.


> We could probably bikeshed a bit on the spec terminology here. "At that
> point the behavior of usb is defined by this specification" seems a little
> wrong in principle to me.  We want web-platform-tests to be testing the Web
> USB spec itself - so test cases should be written against the algorithms
> defined there (not in a spec for a feature that technically doesn't ship to
> users).  But since the testing spec then refers back to running algorithms
> in the Web USB spec maybe that's OK?
>

I am open to suggestions on the best language to use here. In general I
want the WebUSB Testing API specification to be not be replacing algorithms
in the WebUSB API specification but to define the results of actions the
WebUSB API requires the UA to perform on the device or host system, such as
asking the user for permission or reading and writing data to the device.