Re: UserAgent-specific files in Web Platform Tests

We need to figure this out for lots of specs now, and I think the approach
taken makes a lot of sense: specs simply define the APIs that are needed to
test them, it's not somebody else's problem.

However, I would like to go a bit further and treat this more like we treat
any bit of API that tests define. Tests should simply assume that the APIs
exist, and otherwise fail. Having stubs of the APIs could make test
failures more explicit, but it seems like we could do without them. It
could be something like:

async_test(t => {
  navigator.bluetooth.test.setLEAvailability(false);
  // and so on
});

If lots of tests need the same setup, one can of course put that in a
shared bluetooth.js that fails more gracefully than the above one-liner.

In order to actually make the test APIs available Chromium might need to do
some things in its testharnessreport.js, or perhaps provide a command-line
flag if we can figure out how to make it work for vanilla Chrome builds. In
any case, web-platform-tests would just assume their presence.

Would that work?

Since we're trying to come up with a solution that can be copy-pasted into
other areas, there is the question of a namespace. One approach is to just
say that all specs are free to put their testing APIs wherever they like.
To be specified using Web IDL, if not implemented that way, it might end up
requiring a [Testing] extended attribute so that it's clear what things are
for testing only.

Another approach which I've argued for is to have a testing namespace, and
that all specs would put their testing stuff in a "partial namespace
testing", leaving us with a single object to expose or not expose.

At this point, I'm inclined to say we should *not* enforce a testing
namespace, and just see what people end up doing organically. As long as
the APIs are only used in web-platform-tests, making changes to harmonize
after the fact will be possible, if so desired.

Feedback from non-Chromium folks much appreciated :)

On Thu, Mar 16, 2017 at 5:26 AM Vincent Scheib <scheib@chromium.org> wrote:

> +1
>
> stubs sounds good, and if possible the stubs would throw an assert
> pointing to instructions regarding how the platform-fakes files are
> intended to be replaced with implementations.
>
> On Wed, Mar 15, 2017 at 11:42 AM, Reilly Grant <reillyg@chromium.org>
> wrote:
>
> I would like to try formally specifying this for WebUSB as well.
>
> On Thu, Mar 9, 2017 at 8:05 PM Matt Giuca <mgiuca@chromium.org> wrote:
>
> I love this approach! Thanks for sharing and the write-up, Gio.
>
> > On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> s/empty/stubs?
>
> I would definitely be up for converting my navigator.share
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/webshare/share-success.html>
> and navigator.getInstalledRelatedApps
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/installedapp/getinstalledrelatedapps.html> layout
> tests (which currently use an explicit mock of calls to the Mojo service)
> to a standard fake interface. Since my APIs are significantly simpler than
> Bluetooth, I might give it a shot and report back to this group. (Note
> though that they aren't standardised yet so I'm not sure if they'd be
> includeable in TestHarness. Still would serve as a useful case study.)
>
> On Fri, 10 Mar 2017 at 14:52 Giovanni Ortuño <ortuno@chromium.org> wrote:
>
> Hi all,
>
> Some context: We, the Web Bluetooth team, are looking into upstreaming our
> Chromium Layout Tests to Web Platform Tests. In order to test the Web
> Bluetooth API, we are introducing a Test API that accompanies the spec and
> allows our tests to fake Bluetooth Devices: Web Bluetooth Test
> <https://docs.google.com/document/d/1Nhv_oVDCodd1pEH_jj9k8gF4rPGb_84VYaZ9IG8M_WY/edit#heading=h.ap8dnjfog4qc>
> .
>
> Parts of this API are implemented in JS. These parts are Chromium
> specific, e.g. how to talk with our IPC system, so it wouldn't make sense
> to include them as resources.
>
> To that extent, we would like to add a file called "web-bluetooth-test.js"
> which would be similar to "testharnessreport.js" to the testharness repo.
> On the main repo that file would be empty but on the Chromium repo that
> file would have the necessary code to fake devices in Chromium.
>
> There are many APIs that follow a similar pattern: they define a Test API
> surface that they use to fake behavior. Some examples include Geolocation
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/geolocation-api/error.html?type=cs&q=mojo-helpers+file:%5Esrc/third_party/WebKit/LayoutTests/geolocation-api/+package:%5Echromium$&l=17>,
> Vibration
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vibration/vibration-durations.html?l=13>,
> NFC
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/nfc/push.html?l=73>,
> Sensors
> <https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/sensor/accelerometer.html?l=45>,
> etc. So we think it would make sense to add a folder to include all of
> these Test APIs in, straw-man proposal: platform-fakes.
>
> ./
> ./testharness.js
> ./testharnessreport.js
> ./platform-fakes/web-bluetooth-test.js
> ./platform-fakes/geolocation-test.js
> ...
>
> Do y'all think this is a good approach?
>
> Let me know what you think,
>
> Gio
>
>
>