Re: Specifications with matching test APIs

On Mon, Apr 24, 2017 at 4:33 PM James Graham <james@hoppipolla.co.uk> wrote:

> On 22/04/17 02:28, Reilly Grant wrote:
> > On Fri, Apr 21, 2017 at 6:11 PM Rick Byers <rbyers@chromium.org> wrote:
> >
> >> Thanks Reilly, I support continuing to move in this direction!
> >>
> >> Are you planning in shipping this API in release Chrome builds behind a
> >> flag?  I think it would be reasonable to modify the WPT infrastructure
> >> (stability_checker, dashboard) to pass a --enable-testing-apis flag
> >> (although we might need to consider the security implications of that
> for
> >> the WPT infrastructure running potentially untrusted test patches).
> But I
> >> don't think we'd want to use content_shell (or even Chromium builds) in
> >> that infrastructure - at least not in place of Chrome builds.
> >>
> >
> > The ability to override the Mojo services provided to the renderer from
> the
> > renderer itself (which is how Chromium's polyfill for this API is
> > implemented) is only available in content_shell when the
> --run-layout-test
> > parameter is passed. There have been discussions with the Mojo team about
> > making this available in production Chrome builds when a flag is enabled.
> > It would have to be a flag which displays the "unsupported flag, security
> > and stability will suffer" infobar because it effectively allows
> arbitrary
> > JavaScript to run with the privileges of the renderer.
>
> This seems like:
>
> * An API with unclear vendor buy-in.
> * An test API that is (effectively) not available outside Chromium CI
> (although it's possible to get contentshell builds it's not a bad
> approximation to assume that no one will).
>
> So I'm pretty worried about this approach. It seems like there's a high
> chance that the test api will encode Blink implementation details, we
> will struggle to run the tests outside your CI, and web developers who
> want to test their USB-using website will be left to search for a
> different solution.
>

I think it's probably true that there are ways of depending on
implementation details with a testing API that are unlikely/impossible
using a WebDriver extension, but with either approach it would be very
surprising if the tests didn't need any adjustment when the second
implementer starts running them. As long as the effort to find and fix such
problems is much lower than writing tests from scratch, is it not still a
net win? It seems likely to me that this would be the case, especially when
making a conscious effort to define a testing API that doesn't depend on
implementation details.

Until there is a second implementer showing public interest, would it help
to contain tests like this to webusb/chromium/ or similar?