Re: WPT test dependencies

Am 30.11.2017 um 18:43 schrieb Bernard Aboba:
> The web-platform-tests dashboard (see: https://wpt.fyi/webrtc ) contains a disclaimer that it does "not contain useful metrics for evaluation or comparison of web platform features".
> 
> That said, the RTCIceTransport.html test does not succeed even on browsers that support the RTCIceTransport (e.g. Safari).
> 
> The test is failing because this test (and the RTCDtlsTransport test ) depend on the SctpTransport object:
> 
>      "ReferenceError: RTCSctpTransport is not defined"
> 
> Why does the RTCIceTransport test depend on the RTCSctpTransport being defined?
> 
> For example, if a browser supports RtpSender/Receiver, DtlsTransport and IceTransport but not SctpTransport couldn't an RTCIceTransport object be obtained from sender.transport.transport ?

So the second page here have a couple of problems:
     https://wpt.fyi/webrtc/RTCConfiguration-bundlePolicy.html

claims Chrome does not support a balanced bundlePolicy or max-bundle 
which is not true from my experience. The test says:
   new RTCPeerConnection({ bundlePolicy: 'balanced' }) should succeed

but attempts to use getConfiguration. The idiomatically correct test for 
this is to test that
     new RTCPeerConnection({bundlePolicy: 'balanced'})

does not throw. The current test fails for unrelated reasons. It also 
fails in Edge because Edge's native does not like being called without 
iceServers.

None of this is mentioned in the review of the pull request.
     https://github.com/w3c/web-platform-tests/pull/6478
which makes me concerned about the review process.

A good test for bundlePolicy is *much* harder to write. It has to make 
assertions about the type of ice candidates emitted.


This problem of test style persists on a lot of pages:
     https://wpt.fyi/webrtc/RTCConfiguration-iceCandidatePoolSize.html
     https://wpt.fyi/webrtc/RTCConfiguration-iceServers.html
     https://wpt.fyi/webrtc/RTCConfiguration-iceTransportPolicy.html
     https://wpt.fyi/webrtc/RTCConfiguration-rtcpMuxPolicy.html

In the last set of tests Firefox fails because of the lack of 
setConfiguration in the constructor.

I did not bother to look much further but randomly poking around found 
this gem:
 
https://wpt.fyi/webrtc/RTCPeerConnection-createOffer-offerToReceive.html

fails in Edge because its native RTCPeerConnection does a silly thing 
Chrome used to do and throw on new RTCPeerConnection(). Again, trivial 
fix, just call new RTCPeerConnection(null)


Writing good tests is hard. There are some quirks required to make them 
work in all browsers, the current approach of failing without need is 
not useful at all.

Received on Wednesday, 31 January 2018 06:35:35 UTC