Re: UserAgent-specific files in Web Platform Tests

Hi Mike,

Just to add something to your last paragraph. I had this doc
<https://docs.google.com/document/d/1fbkVn0_8CD0cb8TE8SbEHU8I8PZD6v58gVclNeMKrUQ/edit>
 prepared back then for the input injection and compared the two method of
using WebDriver or a browser exposed API.

Particularly the desire for having an alternative way of injecting input
rather than only relying on WebDriver is to be able to exchange the
isolated property of WebDriver for efficiency of injecting input. So one
can more easily run thousands of manual tests on each and every commit. At
the end for example WebDriverJs can be an implementation of that testing
API (rather than the only testing API) and as long as the test page is
using that API they can easily use either the WebDriver path or to choose
browser exposed path if that is exposed in the browser and if they are
interested in a better performance.

Cheers,
Navid

On Mon, Apr 3, 2017 at 10:43 AM Mike Pennisi <mike@bocoup.com> wrote:

> I'm wondering about extending WebDriver for testing internals. By using its
> built-in "Protocol Extensions" mechanism [1], each specification could
> maintain
> its own definition of the API it required.
>
> This was discussed at the "Web Platform Test Integration Convergence"
> meeting
> in January. The minutes from that meeting [2] suggest some additional
> considerations that I may not fully appreciate:
>
> > Security limitation: webdriver cannot do more things than the user can.
> > Principle is that a compromised browser shouldn't have any additional
> > ability.
>
> It's not clear why this is taken as a principle, especially in light of the
> discussion here (i.e. a JavaScript API that can do more things than the
> user
> can).
>
> > WebDriver would commit us to not using sync testing
>
> I believe this is related to the fact that WebDriver is an HTTP-based
> protocol,
> so if tests were written from with the browser, they would need to be built
> from asynchronous Fetch requests. I'm not sure if this is a problem,
> though.
> Tests could be expressed with out-of-process scripts where synchronous HTTP
> communication is possible. Even within the browser, support for async
> functions
> is improving rapidly [3], and this greatly reduces the cognitive overhead
> usually associated with asynchronous programming in the browser.
>
> In my mind, one of the major considerations here is security. The behaviors
> we're trying to automate are taken for granted by end users. If this
> functionality were mistakenly enabled on the web at large, the "User agent"
> trust model would break down. It seems critical that these testing
> capabilities
> are only enabled in the most specific circumstances, and that they are
> otherwise isolated from the web platform at large.
>
> My concern is that if we design this functionality *too* organically, our
> solution will be optimized for developer ergonomics and architectural
> simplicity, and that this will be more susceptible to security
> vulnerabilities.
>
> I've read some limited criticism about the scalability of a
> WebDriver-backed
> solution for platform testing [4], but the traits that increase overhead
> from
> an efficiency standpoint (e.g. "a lot of communication across difference
> processes") are the same traits that would discourage accidental
> deployment.
>
> I'm also interested in consolidating efforts more generally; WebDriver is
> just
> entering the CR phase, but it is a well-established project that was
> created
> to address many of the needs we're discussing here. When I consider the use
> cases for WebDriver and the use cases for the kind of JavaScript testing
> API
> that's being discussed here, I see a lot of overlap. I may be missing a
> distinction, though. Are their differences in the use cases? Or would a
> pure
> JavaScript testing API make WebDriver obsolete?
>
> This is a really important issue, so kudos to the Bluetooth team (and
> everyone
> else involved) for starting the discussion early!
>
> [1] W3C WebDriver working draft, "Protocol Extensions"
>     https://www.w3.org/TR/webdriver/#protocol-extensions
> [2] "Web Platform Test Integration Convergence"
>
> https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE
> [3] caniuse.com, "Async Functions"
>     http://caniuse.com/#feat=async-functions
> [4] "Input Automation in WPT Repo"
>
> https://docs.google.com/document/d/1fbkVn0_8CD0cb8TE8SbEHU8I8PZD6v58gVclNeMKrUQ/edit#heading=h.59vkrk26forv
>
> On 03/29/2017 11:08 AM, Philip Jägenstedt wrote:
>
> We need to figure this out for lots of specs now, and I think the approach
> taken makes a lot of sense: specs simply define the APIs that are needed to
> test them, it's not somebody else's problem.
>
> However, I would like to go a bit further and treat this more like we
> treat any bit of API that tests define. Tests should simply assume that the
> APIs exist, and otherwise fail. Having stubs of the APIs could make test
> failures more explicit, but it seems like we could do without them. It
> could be something like:
>
> async_test(t => {
>   navigator.bluetooth.test.setLEAvailability(false);
>   // and so on
> });
>
> If lots of tests need the same setup, one can of course put that in a
> shared bluetooth.js that fails more gracefully than the above one-liner.
>
> In order to actually make the test APIs available Chromium might need to
> do some things in its testharnessreport.js, or perhaps provide a
> command-line flag if we can figure out how to make it work for vanilla
> Chrome builds. In any case, web-platform-tests would just assume their
> presence.
>
> Would that work?
>
> Since we're trying to come up with a solution that can be copy-pasted into
> other areas, there is the question of a namespace. One approach is to just
> say that all specs are free to put their testing APIs wherever they like.
> To be specified using Web IDL, if not implemented that way, it might end up
> requiring a [Testing] extended attribute so that it's clear what things are
> for testing only.
>
> Another approach which I've argued for is to have a testing namespace, and
> that all specs would put their testing stuff in a "partial namespace
> testing", leaving us with a single object to expose or not expose.
>
> At this point, I'm inclined to say we should *not* enforce a testing
> namespace, and just see what people end up doing organically. As long as
> the APIs are only used in web-platform-tests, making changes to harmonize
> after the fact will be possible, if so desired.
>
> Feedback from non-Chromium folks much appreciated :)
>
> On Thu, Mar 16, 2017 at 5:26 AM Vincent Scheib <scheib@chromium.org>
> wrote:
>
> +1
>
> stubs sounds good, and if possible the stubs would throw an assert
> pointing to instructions regarding how the platform-fakes files are
> intended to be replaced with implementations.
>
> On Wed, Mar 15, 2017 at 11:42 AM, Reilly Grant <reillyg@chromium.org>
> wrote:
>
> I would like to try formally specifying this for WebUSB as well.
>
> On Thu, Mar 9, 2017 at 8:05 PM Matt Giuca <mgiuca@chromium.org> wrote:
>
> I love this approach! Thanks for sharing and the write-up, Gio.
>
> > On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> s/empty/stubs?
>
> I would definitely be up for converting my navigator.share
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/webshare/share-success.html>
> and navigator.getInstalledRelatedApps
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/installedapp/getinstalledrelatedapps.html> layout
> tests (which currently use an explicit mock of calls to the Mojo service)
> to a standard fake interface. Since my APIs are significantly simpler than
> Bluetooth, I might give it a shot and report back to this group. (Note
> though that they aren't standardised yet so I'm not sure if they'd be
> includeable in TestHarness. Still would serve as a useful case study.)
>
> On Fri, 10 Mar 2017 at 14:52 Giovanni Ortuño <ortuno@chromium.org> wrote:
>
> Hi all,
>
> Some context: We, the Web Bluetooth team, are looking into upstreaming our
> Chromium Layout Tests to Web Platform Tests. In order to test the Web
> Bluetooth API, we are introducing a Test API that accompanies the spec and
> allows our tests to fake Bluetooth Devices: Web Bluetooth Test
> <https://docs.google.com/document/d/1Nhv_oVDCodd1pEH_jj9k8gF4rPGb_84VYaZ9IG8M_WY/edit#heading=h.ap8dnjfog4qc>
> .
>
> Parts of this API are implemented in JS. These parts are Chromium
> specific, e.g. how to talk with our IPC system, so it wouldn't make sense
> to include them as resources.
>
> To that extent, we would like to add a file called "web-bluetooth-test.js"
> which would be similar to "testharnessreport.js" to the testharness repo.
> On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> There are many APIs that follow a similar pattern: they define a Test API
> surface that they use to fake behavior. Some examples include Geolocation
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/geolocation-api/error.html?type=cs&q=mojo-helpers+file:%5Esrc/third_party/WebKit/LayoutTests/geolocation-api/+package:%5Echromium$&l=17>,
> Vibration
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vibration/vibration-durations.html?l=13>,
> NFC
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/nfc/push.html?l=73>,
> Sensors
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/sensor/accelerometer.html?l=45>,
> etc. So we think it would make sense to add a folder to include all of
> these Test APIs in, straw-man proposal: platform-fakes.
>
> ./
> ./testharness.js
> ./testharnessreport.js
> ./platform-fakes/web-bluetooth-test.js
> ./platform-fakes/geolocation-test.js
> ...
>
> Do y'all think this is a good approach?
>
> Let me know what you think,
>
> Gio
>
>
>
>

Received on Monday, 3 April 2017 14:59:33 UTC