RE: WPT test dependencies

Fippo said: 

"claims Chrome does not support a balanced bundlePolicy or max-bundle
which is not true from my experience. The test says:
   new RTCPeerConnection({ bundlePolicy: 'balanced' }) should succeed

but attempts to use getConfiguration. The idiomatically correct test for
this is to test that
     new RTCPeerConnection({bundlePolicy: 'balanced'})

does not throw. The current test fails for unrelated reasons."

[BA] In general, we should try to have focused tests with as few dependencies as possible. This makes the
outcome of the tests easier to understand. 

For example, we could have one test for whether new RTCPeerConnection({bundlePolicy: 'balanced'}) throws,
and another test of getConfiguration.

Fippo also said: 

"A good test for bundlePolicy is *much* harder to write. It has to make
assertions about the type of ice candidates emitted."

[BA] Right - this is true for many other tests as well.  We can test whether the functionality is supported at a basic level, but then there is the question of whether the functionality is *correctly* supported, which is harder, particularly if protocol compliance or interoperability is involved.  

Fippo said: 

"Writing good tests is hard. There are some quirks required to make them work in all browsers, the current approach of failing without need is not useful at all."

[BA] Agreed.  While the test suite (correctly) says that the results cannot be used to evaluate browsers, the results would be a lot more useful if we could easily correlate failures to specific missing or incorrectly implemented functionality.  As it is, to figure out what is missing requires going over every test to dis-entangle the dependencies and figure out what actually caused the test to fail.  

In particular, this makes it very difficult to evaluate whether a particular aspect of the specification has been implemented by multiple browsers.  It seems to me that that goal would require more focussed tests.

Received on Wednesday, 31 January 2018 17:06:30 UTC