Re: Specifications with matching test APIs

Support for using the polyfill in stable Chrome landed more quickly than I
anticipated. This patch, moving Blink's LayoutTests for WebUSB into
web-platform-tests will be landing soon:

https://codereview.chromium.org/2789723003/

Running Chrome with --enable-blink-features=MojoJS,MojoJSTest should allow
the tests to pass in any recent build of Chrome. I've tested this out
locally and am excited to try it out on w3c-test.org.

On Wed, Jun 14, 2017 at 1:24 AM Philip Jägenstedt <foolip@google.com> wrote:

> Thanks for summarizing, Vincent,
>
> https://wicg.github.io/webusb/test/ is, I believe, the most well-defined
> such API so far. Perhaps a next good step would be to prepare a PR that
> adds one such test to
> https://github.com/w3c/web-platform-tests/tree/master/webusb to make it
> all very concrete, to see what issues come up?
>
> On Tue, Jun 13, 2017 at 7:52 PM Vincent Scheib <scheib@google.com> wrote:
>
>> Related to this thread "Specifications with matching test APIs"
>> <https://www.w3.org/Search/Mail/Public/search?keywords=&hdr-1-name=subject&hdr-1-query=%22Specifications+with+matching+test+APIs%22&index-grp=Public_FULL&index-type=g&type-index=>
>> and the earlier thread "UserAgent-specific files in Web Platform Tests"
>> <https://www.w3.org/Search/Mail/Public/search?keywords=&hdr-1-name=subject&hdr-1-query=%22UserAgent-specific+files+in+Web+Platform+Tests%22&index-grp=Public_FULL&index-type=g&type-index=>
>>
>> This is a summary of a call discussing web platform testing with “Test
>> APIs” including James Graham (Mozilla) and Philip Jägenstedt, Reilly Grant,
>> Vincent Scheib (Google).
>>
>> Web Bluetooth, WebUSB, and WebVR are actively seeking to develop web
>> platform tests with new requirements compared to existing tests. Complex
>> state must be configured for these features to be tested, e.g. fake
>> bluetooth devices. “Test APIs” are proposed as being paired with
>> specifications.
>>
>> Concerns raised and discussed include:
>>
>> A) With new testing patterns we may develop unexpected influence on
>> future implementations. Particularly risky when only a small number of
>> developers are influencing the API. E.g. WebUSB which doesn’t yet have
>> other implementations.
>>
>> B) WebDriver already operates at a high level, helping avoid test APIs
>> being too implementation specific.
>>
>> C) Test APIs that don’t work with browser’s standard shipping versions
>> may be a problem. E.g. they won’t run with web-developer focused testing
>> infrastructure, BrowserStack, Sauce, etc. And, they’re harder to run
>> manually.
>>
>> D) Our primary goal is to have conforming web browser implementations.
>> Considering the goal of “Testing APIs should work for web-app developers”
>> is lower priority.
>>
>> Recommendations:
>>
>> Test APIs should clearly indicate their purpose and scope:
>>
>> - They are intended only for WPT. This means test APIs can be modified
>> more freely if a later implementation discovers limitations in the testing
>> API. We do not want additional resistance to API change due to e.g. web app
>> developers using the test API.
>>
>> - Test APIs should be designed carefully to only use concepts from the
>> feature being tested, and not the details of the implementation. E.g.
>> garbage collection, implementation specific feature details, synchronous
>> responses, etc.
>>
>> Full notes
>> <https://docs.google.com/document/d/1CWiBgegF25yWDgUFYel5hJoRufM0q2uTHcvxzeRxKvY/edit?pli=1#heading=h.qw3pxi7awcst>
>> .
>>
>> On Wed, May 3, 2017 at 8:55 AM, Philip Jägenstedt <foolip@google.com>
>> wrote:
>>
>>> On Mon, Apr 24, 2017 at 4:33 PM James Graham <james@hoppipolla.co.uk>
>>> wrote:
>>>
>>>> On 22/04/17 02:28, Reilly Grant wrote:
>>>> > On Fri, Apr 21, 2017 at 6:11 PM Rick Byers <rbyers@chromium.org>
>>>> wrote:
>>>> >
>>>> >> Thanks Reilly, I support continuing to move in this direction!
>>>> >>
>>>> >> Are you planning in shipping this API in release Chrome builds
>>>> behind a
>>>> >> flag?  I think it would be reasonable to modify the WPT
>>>> infrastructure
>>>> >> (stability_checker, dashboard) to pass a --enable-testing-apis flag
>>>> >> (although we might need to consider the security implications of
>>>> that for
>>>> >> the WPT infrastructure running potentially untrusted test patches).
>>>> But I
>>>> >> don't think we'd want to use content_shell (or even Chromium builds)
>>>> in
>>>> >> that infrastructure - at least not in place of Chrome builds.
>>>> >>
>>>> >
>>>> > The ability to override the Mojo services provided to the renderer
>>>> from the
>>>> > renderer itself (which is how Chromium's polyfill for this API is
>>>> > implemented) is only available in content_shell when the
>>>> --run-layout-test
>>>> > parameter is passed. There have been discussions with the Mojo team
>>>> about
>>>> > making this available in production Chrome builds when a flag is
>>>> enabled.
>>>> > It would have to be a flag which displays the "unsupported flag,
>>>> security
>>>> > and stability will suffer" infobar because it effectively allows
>>>> arbitrary
>>>> > JavaScript to run with the privileges of the renderer.
>>>>
>>>> This seems like:
>>>>
>>>> * An API with unclear vendor buy-in.
>>>> * An test API that is (effectively) not available outside Chromium CI
>>>> (although it's possible to get contentshell builds it's not a bad
>>>> approximation to assume that no one will).
>>>>
>>>> So I'm pretty worried about this approach. It seems like there's a high
>>>> chance that the test api will encode Blink implementation details, we
>>>> will struggle to run the tests outside your CI, and web developers who
>>>> want to test their USB-using website will be left to search for a
>>>> different solution.
>>>>
>>>
>>> I think it's probably true that there are ways of depending on
>>> implementation details with a testing API that are unlikely/impossible
>>> using a WebDriver extension, but with either approach it would be very
>>> surprising if the tests didn't need any adjustment when the second
>>> implementer starts running them. As long as the effort to find and fix such
>>> problems is much lower than writing tests from scratch, is it not still a
>>> net win? It seems likely to me that this would be the case, especially when
>>> making a conscious effort to define a testing API that doesn't depend on
>>> implementation details.
>>>
>>> Until there is a second implementer showing public interest, would it
>>> help to contain tests like this to webusb/chromium/ or similar?
>>>
>>
>>