Re: UserAgent-specific files in Web Platform Tests

I put together a patch demonstrating this idea as it applies to WebUSB's
existing LayoutTests: https://codereview.chromium.org/2775323003/

The file webusb-test.js polyfils navigator.usb.test and would be either
integrated into testharnessreport.js or placed in a nearby directory that
is treated similarly. Since all tests need to call
navigator.usb.test.initialize() first to set up the mocks I've integrated
that into a usb_test() function defined in usb-helpers.js. This function
could be set up to fail gracefully if that function doesn't exist.

On Wed, Mar 29, 2017 at 8:08 AM Philip Jägenstedt <foolip@chromium.org>
wrote:

> We need to figure this out for lots of specs now, and I think the approach
> taken makes a lot of sense: specs simply define the APIs that are needed to
> test them, it's not somebody else's problem.
>
> However, I would like to go a bit further and treat this more like we
> treat any bit of API that tests define. Tests should simply assume that the
> APIs exist, and otherwise fail. Having stubs of the APIs could make test
> failures more explicit, but it seems like we could do without them. It
> could be something like:
>
> async_test(t => {
>   navigator.bluetooth.test.setLEAvailability(false);
>   // and so on
> });
>
> If lots of tests need the same setup, one can of course put that in a
> shared bluetooth.js that fails more gracefully than the above one-liner.
>
> In order to actually make the test APIs available Chromium might need to
> do some things in its testharnessreport.js, or perhaps provide a
> command-line flag if we can figure out how to make it work for vanilla
> Chrome builds. In any case, web-platform-tests would just assume their
> presence.
>
> Would that work?
>
> Since we're trying to come up with a solution that can be copy-pasted into
> other areas, there is the question of a namespace. One approach is to just
> say that all specs are free to put their testing APIs wherever they like.
> To be specified using Web IDL, if not implemented that way, it might end up
> requiring a [Testing] extended attribute so that it's clear what things are
> for testing only.
>
> Another approach which I've argued for is to have a testing namespace, and
> that all specs would put their testing stuff in a "partial namespace
> testing", leaving us with a single object to expose or not expose.
>
> At this point, I'm inclined to say we should *not* enforce a testing
> namespace, and just see what people end up doing organically. As long as
> the APIs are only used in web-platform-tests, making changes to harmonize
> after the fact will be possible, if so desired.
>
> Feedback from non-Chromium folks much appreciated :)
>
> On Thu, Mar 16, 2017 at 5:26 AM Vincent Scheib <scheib@chromium.org>
> wrote:
>
> +1
>
> stubs sounds good, and if possible the stubs would throw an assert
> pointing to instructions regarding how the platform-fakes files are
> intended to be replaced with implementations.
>
> On Wed, Mar 15, 2017 at 11:42 AM, Reilly Grant <reillyg@chromium.org>
> wrote:
>
> I would like to try formally specifying this for WebUSB as well.
>
> On Thu, Mar 9, 2017 at 8:05 PM Matt Giuca <mgiuca@chromium.org> wrote:
>
> I love this approach! Thanks for sharing and the write-up, Gio.
>
> > On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> s/empty/stubs?
>
> I would definitely be up for converting my navigator.share
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/webshare/share-success.html>
> and navigator.getInstalledRelatedApps
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/installedapp/getinstalledrelatedapps.html> layout
> tests (which currently use an explicit mock of calls to the Mojo service)
> to a standard fake interface. Since my APIs are significantly simpler than
> Bluetooth, I might give it a shot and report back to this group. (Note
> though that they aren't standardised yet so I'm not sure if they'd be
> includeable in TestHarness. Still would serve as a useful case study.)
>
> On Fri, 10 Mar 2017 at 14:52 Giovanni Ortuño <ortuno@chromium.org> wrote:
>
> Hi all,
>
> Some context: We, the Web Bluetooth team, are looking into upstreaming our
> Chromium Layout Tests to Web Platform Tests. In order to test the Web
> Bluetooth API, we are introducing a Test API that accompanies the spec and
> allows our tests to fake Bluetooth Devices: Web Bluetooth Test
> <https://docs.google.com/document/d/1Nhv_oVDCodd1pEH_jj9k8gF4rPGb_84VYaZ9IG8M_WY/edit#heading=h.ap8dnjfog4qc>
> .
>
> Parts of this API are implemented in JS. These parts are Chromium
> specific, e.g. how to talk with our IPC system, so it wouldn't make sense
> to include them as resources.
>
> To that extent, we would like to add a file called "web-bluetooth-test.js"
> which would be similar to "testharnessreport.js" to the testharness repo.
> On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> There are many APIs that follow a similar pattern: they define a Test API
> surface that they use to fake behavior. Some examples include Geolocation
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/geolocation-api/error.html?type=cs&q=mojo-helpers+file:%5Esrc/third_party/WebKit/LayoutTests/geolocation-api/+package:%5Echromium$&l=17>,
> Vibration
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vibration/vibration-durations.html?l=13>,
> NFC
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/nfc/push.html?l=73>,
> Sensors
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/sensor/accelerometer.html?l=45>,
> etc. So we think it would make sense to add a folder to include all of
> these Test APIs in, straw-man proposal: platform-fakes.
>
> ./
> ./testharness.js
> ./testharnessreport.js
> ./platform-fakes/web-bluetooth-test.js
> ./platform-fakes/geolocation-test.js
> ...
>
> Do y'all think this is a good approach?
>
> Let me know what you think,
>
> Gio
>
>
>