Re: Specifications with matching test APIs

[Resent from the correct email address.]

On Mon, Apr 24, 2017 at 7:33 AM James Graham <james@hoppipolla.co.uk> wrote:

> On 22/04/17 02:28, Reilly Grant wrote:
> > On Fri, Apr 21, 2017 at 6:11 PM Rick Byers <rbyers@chromium.org> wrote:
> >
> >> Thanks Reilly, I support continuing to move in this direction!
> >>
> >> Are you planning in shipping this API in release Chrome builds behind a
> >> flag?  I think it would be reasonable to modify the WPT infrastructure
> >> (stability_checker, dashboard) to pass a --enable-testing-apis flag
> >> (although we might need to consider the security implications of that
> for
> >> the WPT infrastructure running potentially untrusted test patches).
> But I
> >> don't think we'd want to use content_shell (or even Chromium builds) in
> >> that infrastructure - at least not in place of Chrome builds.
> >>
> >
> > The ability to override the Mojo services provided to the renderer from
> the
> > renderer itself (which is how Chromium's polyfill for this API is
> > implemented) is only available in content_shell when the
> --run-layout-test
> > parameter is passed. There have been discussions with the Mojo team about
> > making this available in production Chrome builds when a flag is enabled.
> > It would have to be a flag which displays the "unsupported flag, security
> > and stability will suffer" infobar because it effectively allows
> arbitrary
> > JavaScript to run with the privileges of the renderer.
>
> This seems like:
>
> * An API with unclear vendor buy-in.
> * An test API that is (effectively) not available outside Chromium CI
> (although it's possible to get contentshell builds it's not a bad
> approximation to assume that no one will).
>
> So I'm pretty worried about this approach. It seems like there's a high
> chance that the test api will encode Blink implementation details, we
> will struggle to run the tests outside your CI, and web developers who
> want to test their USB-using website will be left to search for a
> different solution.
>

In the process of moving from completely Blink-specific tests to something
that can be upstreamed to Web Platform Tests my primary focus has been to
take a critical look at anything about the tests that unintentionally
encodes implementation details. By presenting this as a specification my
hope is to get feedback on any additional issues that I have missed. There
is already a note about the reset()
<https://wicg.github.io/webusb/test/#dom-usbtest-reset> method.

We have a plan to make these tests work outside of our CI by running Chrome
with a single flag that enables the necessary test automation interfaces.

Web developers who want to test a site using WebUSB have other options,
such as a complete fake for the WebUSB API, that require less setup than
what is necessary to test the in-browser implementation.

This is the only option I see for allowing any kind of CI for this type of
API.