Re: WPT test dependencies

On Nov 30, 2017, at 5:41 PM, Nils Ohlmeier <nohlmeier@mozilla.com<mailto:nohlmeier@mozilla.com>> wrote:

I guess this comes down to the ultimate question: is every implementation expected to implement all aspects of the spec?

[BA] It seems to me that a test should test the API it is focussed on, with as few deoendencies as possible. That isn’t a judgement about conformance criteria so much as a desire to easily figure out exactly what is broken when something regresses.

For example, if IceTransport and DtlsTransport depend on SctpTransport, then if SctpTransport regresses (or isn’t yet implemented) the other tests go red even if the IceTransport and DtlsTransport functionality is fine. That seems unhelpful to me, particularly if the end goal is to achieve and maintain a high level of interoperability with a large test matrix, where spurious results could deluge developers with misinformation.