Re: WPT test dependencies

On Fri, Dec 1, 2017 at 9:41 AM, Nils Ohlmeier <nohlmeier@mozilla.com> wrote:
> Just to raise awareness: right now lots of tests are also failing because they require Transceivers to be available even if the test is not testing anything related to Transceivers (because some of the shared helper functions rely on Transceivers).
>
> I guess this comes down to the ultimate question: is every implementation expected to implement all aspects of the spec?

On Fri, Dec 1, 2017 at 12:14 PM, Bernard Aboba
<Bernard.Aboba@microsoft.com> wrote:
> [BA] It seems to me that a test should test the API it is focussed on, with
> as few deoendencies as possible. That isn’t a judgement about conformance
> criteria so much as a desire to easily figure out exactly what is broken
> when something regresses.

I think having as few dependencies is fine, although there is a
question how the dependencies are defined. Do we define the
dependencies as what is in the spec, or in the browser implementers'
road map?

IIRC addTrack is preferred over addTransceiver because Firefox has
implemented addTrack. But Chrome for example had not implemented
addTrack and it is only added recently in nightly. Furthermore Firefox
implementation of addTrack is not exactly conformant to the spec, as
it requires a compulsory mediaStream as second argument. The current
tests in WPT made a compromise to always pass mediaStream to addTrack
to workaround the test dependencies problem. I feel that it is not
very healthy to write code to to accommodate specific browsers only,
but if we had written the tests strictly using the standard addTrack
interface, none of the "higher level" tests would have pass until
either addTransceiver is implemented or addTrack is fixed.

When a feature has few alternative dependency paths, different
browsers would have different preference depending on their
implementation road map. If we want to prefer tests to use a specific
dependency path, it might be helpful to document it as guidelines
somewhere. That way test writers can better plan the test dependencies
and prioritize on features that will be implemented sooner.


Just my 2 cents,

Soares

Received on Monday, 4 December 2017 10:35:44 UTC