Re: WPT test dependencies

When I first read the mail, I was quite in agreement with Philipp, but 
the more I think about it, the less I am.. :)

Regarding reducing dependencies with other parts of the spec, i think it 
is a good goal, and we should do it if the alternative provides the same 
validity of the result. If the alternative is parsing the SDP or not 
testing it, then I would strongly oppose to that.

I agree that most of the test in WPT test the js api layer, like 
validating that if I set a value, then I get the same value back, and 
not the effect of that setting, but in order to test that part we would 
need some kind of external server that test IETF stuff compliance. I 
found that piece missing and really important (for testing simulcast for 
example), although I am not sure if that testing should be in the scope 
of the WPT or on something different (KITE?).

Regarding changing new RTCPeerConnection() by new 
RTCPeerConnection(null) just because some browser does "a silly thing" 
and is not spec compliant, well.. We should not hack the test so a 
certain browser can pass it, but viceversa.

One last comment, I think we should be more careful when making changes 
to the spec that potentially breaks WPTs and if not providing a change 
to the WPT itself, at least open an issue so we can track those changes.

Best regards
Sergio

On 31/01/2018 7:34, Philipp Hancke wrote:
> Am 30.11.2017 um 18:43 schrieb Bernard Aboba:
>> The web-platform-tests dashboard (see: https://wpt.fyi/webrtc ) 
>> contains a disclaimer that it does "not contain useful metrics for 
>> evaluation or comparison of web platform features".
>>
>> That said, the RTCIceTransport.html test does not succeed even on 
>> browsers that support the RTCIceTransport (e.g. Safari).
>>
>> The test is failing because this test (and the RTCDtlsTransport test 
>> ) depend on the SctpTransport object:
>>
>>      "ReferenceError: RTCSctpTransport is not defined"
>>
>> Why does the RTCIceTransport test depend on the RTCSctpTransport 
>> being defined?
>>
>> For example, if a browser supports RtpSender/Receiver, DtlsTransport 
>> and IceTransport but not SctpTransport couldn't an RTCIceTransport 
>> object be obtained from sender.transport.transport ?
>
> So the second page here have a couple of problems:
>     https://wpt.fyi/webrtc/RTCConfiguration-bundlePolicy.html
>
> claims Chrome does not support a balanced bundlePolicy or max-bundle 
> which is not true from my experience. The test says:
>   new RTCPeerConnection({ bundlePolicy: 'balanced' }) should succeed
>
> but attempts to use getConfiguration. The idiomatically correct test 
> for this is to test that
>     new RTCPeerConnection({bundlePolicy: 'balanced'})
>
> does not throw. The current test fails for unrelated reasons. It also 
> fails in Edge because Edge's native does not like being called without 
> iceServers.
>
> None of this is mentioned in the review of the pull request.
>     https://github.com/w3c/web-platform-tests/pull/6478
> which makes me concerned about the review process.
>
> A good test for bundlePolicy is *much* harder to write. It has to make 
> assertions about the type of ice candidates emitted.
>
>
> This problem of test style persists on a lot of pages:
> https://wpt.fyi/webrtc/RTCConfiguration-iceCandidatePoolSize.html
>     https://wpt.fyi/webrtc/RTCConfiguration-iceServers.html
> https://wpt.fyi/webrtc/RTCConfiguration-iceTransportPolicy.html
>     https://wpt.fyi/webrtc/RTCConfiguration-rtcpMuxPolicy.html
>
> In the last set of tests Firefox fails because of the lack of 
> setConfiguration in the constructor.
>
> I did not bother to look much further but randomly poking around found 
> this gem:
>
> https://wpt.fyi/webrtc/RTCPeerConnection-createOffer-offerToReceive.html
>
> fails in Edge because its native RTCPeerConnection does a silly thing 
> Chrome used to do and throw on new RTCPeerConnection(). Again, trivial 
> fix, just call new RTCPeerConnection(null)
>
>
> Writing good tests is hard. There are some quirks required to make 
> them work in all browsers, the current approach of failing without 
> need is not useful at all.
>

Received on Wednesday, 31 January 2018 08:34:45 UTC