Re: UserAgent-specific files in Web Platform Tests

On Mon, Apr 3, 2017 at 9:42 PM Mike Pennisi <mike@bocoup.com> wrote:

> I'm wondering about extending WebDriver for testing internals. By using its
> built-in "Protocol Extensions" mechanism [1], each specification could
> maintain
> its own definition of the API it required.
>
> This was discussed at the "Web Platform Test Integration Convergence"
> meeting
> in January. The minutes from that meeting [2] suggest some additional
> considerations that I may not fully appreciate:
>
> > Security limitation: webdriver cannot do more things than the user can.
> > Principle is that a compromised browser shouldn't have any additional
> > ability.
>
> It's not clear why this is taken as a principle, especially in light of the
> discussion here (i.e. a JavaScript API that can do more things than the
> user
> can).
>
As indicated in notes, this was a concern/constraint by the Safari team.
It's probably a safe a assumption that JavaScript APIs with the same powers
also wouldn't be shipped in the release binary of Safari.

However, this is an issue I think ought not have much impact on what path
is taken. It could be that some browsers will only be possible to test
using special builds, some may require a command line, etc.. I would like
for it to be po ssible to test run all of web-platform-tests using the
binaries that are distributed to users, but *any* way of automating tests
of things like Web Bluetooth would be a huge improvement.

> > WebDriver would commit us to not using sync testing
>
> I believe this is related to the fact that WebDriver is an HTTP-based
> protocol,
> so if tests were written from with the browser, they would need to be built
> from asynchronous Fetch requests. I'm not sure if this is a problem,
> though.
> Tests could be expressed with out-of-process scripts where synchronous HTTP
> communication is possible. Even within the browser, support for async
> functions
> is improving rapidly [3], and this greatly reduces the cognitive overhead
> usually associated with asynchronous programming in the browser.
>
I think this is something that I said. I'm not concerned about the fact
that an async_test or promise_test might be harder to read than a
synchronous test, I think we could probably create nice-enough abstractions
for that to not matter. Rather, it's that we have a lot of testing APIs in
Blink's Internals.idl
<https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/testing/Internals.idl>,
and I'm not confident that all of them would still work as async APIs. Take
effectivePreload() for example, which gets the internal effective preload
state of a media element. If a test gets a media element into a particular
state and then uses an async API, the state may have changed (due to
network activity) by the time it is read. Arguably, we should expose
the effective
preload state <https://github.com/whatwg/html/issues/1932>, but I'm not
confident this will *always* be the right choice.

All this means is that I don't want to commit to using WebDriver for
everything, but each case will have to be argued individually.

> In my mind, one of the major considerations here is security. The behaviors
> we're trying to automate are taken for granted by end users. If this
> functionality were mistakenly enabled on the web at large, the "User agent"
> trust model would break down. It seems critical that these testing
> capabilities
> are only enabled in the most specific circumstances, and that they are
> otherwise isolated from the web platform at large.
>
> My concern is that if we design this functionality *too* organically, our
> solution will be optimized for developer ergonomics and architectural
> simplicity, and that this will be more susceptible to security
> vulnerabilities.
>
> I've read some limited criticism about the scalability of a
> WebDriver-backed
> solution for platform testing [4], but the traits that increase overhead
> from
> an efficiency standpoint (e.g. "a lot of communication across difference
> processes") are the same traits that would discourage accidental
> deployment.
>
I'll refrain from making any certain claims before a testing API of the
sort we're discussion has gone through Chrome's security review and
launched, but on the face of it I don't think there is much difference
between the approaches. In both cases, there are some bits of code that
cause the "API" (WebDriver commands or JavaScript APIs) to be exposed, and
that would have to be controlled by a command line flag or similar.

> I'm also interested in consolidating efforts more generally; WebDriver is
> just
> entering the CR phase, but it is a well-established project that was
> created
> to address many of the needs we're discussing here. When I consider the use
> cases for WebDriver and the use cases for the kind of JavaScript testing
> API
> that's being discussed here, I see a lot of overlap. I may be missing a
> distinction, though. Are their differences in the use cases? Or would a
> pure
> JavaScript testing API make WebDriver obsolete?
>
There certainly is a lot of overlap, which makes this a hard choice.

> This is a really important issue, so kudos to the Bluetooth team (and
> everyone
> else involved) for starting the discussion early!
>
> [1] W3C WebDriver working draft, "Protocol Extensions"
>     https://www.w3.org/TR/webdriver/#protocol-extensions
> [2] "Web Platform Test Integration Convergence"
>
> https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE
> [3] caniuse.com, "Async Functions"
>     http://caniuse.com/#feat=async-functions
> [4] "Input Automation in WPT Repo"
>
> https://docs.google.com/document/d/1fbkVn0_8CD0cb8TE8SbEHU8I8PZD6v58gVclNeMKrUQ/edit#heading=h.59vkrk26forv
>
> On 03/29/2017 11:08 AM, Philip Jägenstedt wrote:
>
> We need to figure this out for lots of specs now, and I think the approach
> taken makes a lot of sense: specs simply define the APIs that are needed to
> test them, it's not somebody else's problem.
>
> However, I would like to go a bit further and treat this more like we
> treat any bit of API that tests define. Tests should simply assume that the
> APIs exist, and otherwise fail. Having stubs of the APIs could make test
> failures more explicit, but it seems like we could do without them. It
> could be something like:
>
> async_test(t => {
>   navigator.bluetooth.test.setLEAvailability(false);
>   // and so on
> });
>
> If lots of tests need the same setup, one can of course put that in a
> shared bluetooth.js that fails more gracefully than the above one-liner.
>
> In order to actually make the test APIs available Chromium might need to
> do some things in its testharnessreport.js, or perhaps provide a
> command-line flag if we can figure out how to make it work for vanilla
> Chrome builds. In any case, web-platform-tests would just assume their
> presence.
>
> Would that work?
>
> Since we're trying to come up with a solution that can be copy-pasted into
> other areas, there is the question of a namespace. One approach is to just
> say that all specs are free to put their testing APIs wherever they like.
> To be specified using Web IDL, if not implemented that way, it might end up
> requiring a [Testing] extended attribute so that it's clear what things are
> for testing only.
>
> Another approach which I've argued for is to have a testing namespace, and
> that all specs would put their testing stuff in a "partial namespace
> testing", leaving us with a single object to expose or not expose.
>
> At this point, I'm inclined to say we should *not* enforce a testing
> namespace, and just see what people end up doing organically. As long as
> the APIs are only used in web-platform-tests, making changes to harmonize
> after the fact will be possible, if so desired.
>
> Feedback from non-Chromium folks much appreciated :)
>
> On Thu, Mar 16, 2017 at 5:26 AM Vincent Scheib <scheib@chromium.org>
> wrote:
>
> +1
>
> stubs sounds good, and if possible the stubs would throw an assert
> pointing to instructions regarding how the platform-fakes files are
> intended to be replaced with implementations.
>
> On Wed, Mar 15, 2017 at 11:42 AM, Reilly Grant <reillyg@chromium.org>
> wrote:
>
> I would like to try formally specifying this for WebUSB as well.
>
> On Thu, Mar 9, 2017 at 8:05 PM Matt Giuca <mgiuca@chromium.org> wrote:
>
> I love this approach! Thanks for sharing and the write-up, Gio.
>
> > On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> s/empty/stubs?
>
> I would definitely be up for converting my navigator.share
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/webshare/share-success.html>
> and navigator.getInstalledRelatedApps
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/installedapp/getinstalledrelatedapps.html> layout
> tests (which currently use an explicit mock of calls to the Mojo service)
> to a standard fake interface. Since my APIs are significantly simpler than
> Bluetooth, I might give it a shot and report back to this group. (Note
> though that they aren't standardised yet so I'm not sure if they'd be
> includeable in TestHarness. Still would serve as a useful case study.)
>
> On Fri, 10 Mar 2017 at 14:52 Giovanni Ortuño <ortuno@chromium.org> wrote:
>
> Hi all,
>
> Some context: We, the Web Bluetooth team, are looking into upstreaming our
> Chromium Layout Tests to Web Platform Tests. In order to test the Web
> Bluetooth API, we are introducing a Test API that accompanies the spec and
> allows our tests to fake Bluetooth Devices: Web Bluetooth Test
> <https://docs.google.com/document/d/1Nhv_oVDCodd1pEH_jj9k8gF4rPGb_84VYaZ9IG8M_WY/edit#heading=h.ap8dnjfog4qc>
> .
>
> Parts of this API are implemented in JS. These parts are Chromium
> specific, e.g. how to talk with our IPC system, so it wouldn't make sense
> to include them as resources.
>
> To that extent, we would like to add a file called "web-bluetooth-test.js"
> which would be similar to "testharnessreport.js" to the testharness repo.
> On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> There are many APIs that follow a similar pattern: they define a Test API
> surface that they use to fake behavior. Some examples include Geolocation
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/geolocation-api/error.html?type=cs&q=mojo-helpers+file:%5Esrc/third_party/WebKit/LayoutTests/geolocation-api/+package:%5Echromium$&l=17>,
> Vibration
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vibration/vibration-durations.html?l=13>,
> NFC
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/nfc/push.html?l=73>,
> Sensors
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/sensor/accelerometer.html?l=45>,
> etc. So we think it would make sense to add a folder to include all of
> these Test APIs in, straw-man proposal: platform-fakes.
>
> ./
> ./testharness.js
> ./testharnessreport.js
> ./platform-fakes/web-bluetooth-test.js
> ./platform-fakes/geolocation-test.js
> ...
>
> Do y'all think this is a good approach?
>
> Let me know what you think,
>
> Gio
>
>
>
>