Re: UserAgent-specific files in Web Platform Tests

On Tue, Apr 4, 2017 at 12:53 AM James Graham <james@hoppipolla.co.uk> wrote:

> On 29/03/17 16:08, Philip Jägenstedt wrote:
> > We need to figure this out for lots of specs now, and I think the
> approach
> > taken makes a lot of sense: specs simply define the APIs that are needed
> to
> > test them, it's not somebody else's problem.
> >
> > However, I would like to go a bit further and treat this more like we
> treat
> > any bit of API that tests define. Tests should simply assume that the
> APIs
> > exist, and otherwise fail. Having stubs of the APIs could make test
> > failures more explicit, but it seems like we could do without them. It
> > could be something like:
> >
> > async_test(t => {
> >   navigator.bluetooth.test.setLEAvailability(false);
> >   // and so on
> > });
> >
> > If lots of tests need the same setup, one can of course put that in a
> > shared bluetooth.js that fails more gracefully than the above one-liner.
>
> If there are clear specs for how these APIs should work that's probably
> OK. But it really has to be specced in just as much detail as any other
> part of the specification (perhaps more since it may be intrinsically
> difficult to test). I think that there's a serious danger here that we
> get tests which depend on some test-only API that has a single
> implementation and not enough detail in the specification and other
> vendors end up unable to make a compatible implemenation of the test API.
>
>
I totally agree here. We recognized this risk when designing the Bluetooth
Test API as well. I think in the end it will be up to the spec owners to
design their Test APIs in a way that can be implemented across all User
Agents and across all platforms, in the same way they design their actual
APIs. In the case of Bluetooth we designed the Test API ignoring our
implementation and focusing on the concepts of the actual Bluetooth API.

For that reason, for the "easy" case here (creating artifical user
> input), I really want us to start with a cross-browser implementation
> based on the WebDriver standard. For cases like bluetooth that don't
> already have an equivalent cross-browser standard test API we should
> strongly consider which parts it makes sense to expose to web developers
> and put those in the WebDriver standard. For the remaining parts great
> care is required to ensure that the test API isn't an afterthought but
> treated with the same care, and level of vendor buy-in, as the API it's
> trying to test.
>
>