Re: UserAgent-specific files in Web Platform Tests

On Tue, May 23, 2017 at 2:37 PM James Graham <james@hoppipolla.co.uk> wrote:

> On 22/05/17 17:13, Philip Jägenstedt wrote:
> > Thanks for sharing that, Attila, I had no idea about this project in
> > Servo, but it's great that the tests provided value.
> >
> > It is actually not too surprising that the tests were copied out of the
> > Chromium tree, something happened with Service Worker tests although in
> > that case they were copied into web-platform-tests and currently there's
> > work ongoing to reconcile them. Over time, something similar might
> > happen with WebBluetooth if there's any friction in keeping the tests in
> > sync.
> >
> > James, I understand that you still have concerns with this approach, and
> > there are things that might go wrong. However, I would suggest that the
> > value of sharing tests is potentially high, and that having forked test
> > suites is no fun. web-platform-tests would be the main test suite for
> > WebBluetooth in Chrome, so just like for the spec itself, there should
> > always be someone who can answer questions on the Chromium side.
> >
> > If we upstream these tests, I think the following would be reasonable:
> >
> >   * A README that clearly documents the requirements for running the
> tests.
> >   * Intelligible failure messages when testing APIs aren't present.
> >   * A Chromium tracking bug
> >     <https://bugs.chromium.org/p/chromium/issues/detail?id=725105> to
> >     get the tests running on vanilla Chrome binaries and working on PR
> >     stability testing and the upcoming wpt dashboard.
> >
> > Do you think that would suffice?
>
> So, I still think this general approach is likely to be problematic in
> the end. But I don't object to upstreaming these tests if there are
> actually multiple implementations using them, subject to completion of
> the above action items, with the proviso that I won't be surprised if we
> revisit this problem in the future and end up deciding that a different
> tack is necessary. In particular I don't think we should use this as a
> precedent for future testsuites to follow; I don't think it's a "good
> enough" solution in general — and especially not in cases where there is
> only a Chrome implementation — but I do think that upstreaming them is
> more helpful than Servo pulling them out of the Blink repo.
>

Thanks James, let's see how it goes. I would also not be surprised if we
need to revisit how this works, after all it is (I think) the first attempt
at mocking a complex API like this.

As for setting a precedent, while I think that this style of testing will
be the right choice some of the time, let's keep discussing on a
case-by-case basis.