Re: WPT test dependencies

On 02/05/2018 07:03 PM, Philipp Hancke wrote:
>> Even though we do not have the equivalent of block chain's smart
>> contract, we can modify the commit / review / merge process to enforce
>> it.
>
> The current process had this gem slip through as well:
>     https://github.com/w3c/web-platform-tests/pull/5847
>
> In this particular case the spec was vague:
>
> https://github.com/w3c/web-platform-tests/pull/5847/files#diff-908cb5d58eaaf36277efb47afe2fe6c6R120
>
> and the "behave as if" resulted in the longest spec issue ever.

Nah - not even close :-)

>
> At the same time the test assertion
>
> https://github.com/w3c/web-platform-tests/pull/5847/files#diff-908cb5d58eaaf36277efb47afe2fe6c6R136
>
> was not passing in Chrome or Firefox or Edge. This should have raised
> a flag. It did not.
>
> That test descriptions like "new RTCIceCandidate()" pass review is
> absurd.
>
>
> A working review process would require to show which browsers a test
> passed in and discuss failures. If a test did not pass the ball goes
> to a person working on that browser for triage. This might result in a
> bug being filed against that browser or the spec.
>
> Such a process is of course much more time-consuming and expensive --
> unless you are trying to make this a "community effort".

The special case of the initial webrtc test suite is a special case:
someone who did not have experience writing WPT tests wrote a bunch of
stuff and sent them in for review, and didn't get adequate feedback
either from test experts or from subject matter experts.

We're much better off having the tests to iterate over than not having
the tests, so I don't think we should be too unhappy about them - still,
we should aim to do better.

>
>
> > Don't be shy, "Be a programmer" (TM).
>
> looks like my last commit in the webrtc directory is more recent than
> yours.
>

Received on Tuesday, 6 February 2018 09:31:45 UTC