Re: UserAgent-specific files in Web Platform Tests

On 29/03/17 16:08, Philip Jägenstedt wrote:
> We need to figure this out for lots of specs now, and I think the approach
> taken makes a lot of sense: specs simply define the APIs that are needed to
> test them, it's not somebody else's problem.
>
> However, I would like to go a bit further and treat this more like we treat
> any bit of API that tests define. Tests should simply assume that the APIs
> exist, and otherwise fail. Having stubs of the APIs could make test
> failures more explicit, but it seems like we could do without them. It
> could be something like:
>
> async_test(t => {
>   navigator.bluetooth.test.setLEAvailability(false);
>   // and so on
> });
>
> If lots of tests need the same setup, one can of course put that in a
> shared bluetooth.js that fails more gracefully than the above one-liner.

If there are clear specs for how these APIs should work that's probably 
OK. But it really has to be specced in just as much detail as any other 
part of the specification (perhaps more since it may be intrinsically 
difficult to test). I think that there's a serious danger here that we 
get tests which depend on some test-only API that has a single 
implementation and not enough detail in the specification and other 
vendors end up unable to make a compatible implemenation of the test API.

For that reason, for the "easy" case here (creating artifical user 
input), I really want us to start with a cross-browser implementation 
based on the WebDriver standard. For cases like bluetooth that don't 
already have an equivalent cross-browser standard test API we should 
strongly consider which parts it makes sense to expose to web developers 
and put those in the WebDriver standard. For the remaining parts great 
care is required to ensure that the test API isn't an afterthought but 
treated with the same care, and level of vendor buy-in, as the API it's 
trying to test.

Received on Monday, 3 April 2017 14:53:24 UTC