Re: On the way to WebRTC 1.0

In the past we have had a “CR” label, meaning the issue had to be resolved for CR.

Dom - Can you respond relating to the PR requirements? In addition to interop demonstration do we get to decide what issues block PR as we did with CR? We might have a “Zero Bug Bounce” but staying at zero is challenging...



On Oct 19, 2018, at 3:25 PM, youenn fablet <yfablet@apple.com<mailto:yfablet@apple.com>> wrote:



On Oct 19, 2018, at 2:43 PM, Bernard Aboba <Bernard.Aboba@microsoft.com<mailto:Bernard.Aboba@microsoft.com>> wrote:

Youenn said:


I thought a bit on how to best finalize WebRTC 1.0 in terms of specification work.
I believe we need to at least do two things:
1. Resolve all WebRTC 1.0 issues

[BA] At TPAC, we have allocated time for discussion of  roughly a baker's dozen open issues on webrtc-pc. The breakdown of issues (45) by label is as follows:

For discussion at TPAC:                          13
Editorial:                                                   8
Question (Issue not yet clear):                 4
Needs submitter/assignee action            4
PR exists                                                  3
Enhancement (e.g. not WebRTC 1.0):       3
Simulcast  non-TPAC                               2
Pending RTCWEB WG Action:                 1
Miscellaneous                                         7

2. Prove (to ourselves and others) that the WebRTC 1.0 spec is implementable in an interoperable manner

[BA] At TPAC we will be having discussions relating to conformance testing (WPT) as well as protocol interoperability testing (KITE).


For 1, it might be a healthy exercise for the Working Group to label issues as "WebRTC 1.0” vs. “Post WebRTC 1.0”.

[BA] The "Enhancement" label is applied to Issues that will not be handled within the WebRTC-PC specification.

Doe that mean that all issues that are not labelled as “Enhancement” should be resolved to go to PR?
I see some issues labelled as Editorial, IceBox, Later or ’Needs submitter action' for instance.
Also, is CSP a prerequisite for WebRTC 1.0 PR?

Would it help to have an explicit label?

Since the spec will continue evolving, we might also want to take some explicit "test step" for any PR being made so that we can classify each change as:
- Editorial/No need for test
- Need for WPT test (and a link to the corresponding WPT test PR)
- Need for manual test/investigation or KITE test

[BA] We have had a "test before commit" policy in place since TPAC 2017.  We will be going over how well this process is working at TPAC.  Overall, I would note that areas of the specification that are most problematic to implementers are have so far have been difficult to test with WPT  (e.g. simulcast). We will be talking about approaches we might use to address this.

Sounds good.
The thing I want to avoid is to again and again go through the spec and make sure we have full coverage.
If at a given time, we know we have full coverage and following on that, we keep track of whether any new PR requires a test, we would keep a good full coverage proof at a limited cost.


At the end of the day, when we need to prove interoperability, we should be able to show:
- A WPT report showing that each WPT test is passing in at least two browsers (or we have explanations on particular failures).

[BA] WPT tests API conformance, not protocol interoperability. Note that  in a number of cases, the results of the WPT tests do not reflect the true status because the tests have dependencies orthogonal to the functionality being tested that cause them to fail.  For example, there are tests that rely on getUserMedia which will fail if the prompt cannot automatically be dismissed, even if they would pass if a Track could be provided by other means (e.g. generated by WebAudio).  So things are not quite as bad as the WPT test matrix would make them out to be. The confluence matrix (which shows which APIs are implemented) appears more "green", for example.

Agreed, which is why a straight WPT report cannot be exploited as is.
Also this is clearly not great as WebRTC maturity advertisement.

getUserMedia can be easily fixed by running WPT ourselves through WPT test runner and setting persistent access (or in Safari through web driver), ditto for ICE candidate filtering for instance.
Maybe some non browser WebRTC SDKs could also be used to show additional support of some constructs?

Also, some tests might fail due to differences that are not meaningful to WebRTC 1.0.
It is fine excluding these tests as long as we keep a rationale.


[Youenn] Given that we are targeting the WebRTC 1.0 spec, I would believe that WPT should be our topmost priority.

[BA] I would agree that testing overall is very important (not just WPT but also KITE).  I am looking forward to a discussion of how we might improve the process and raise the level of engagement so as to make better progress.

Note that I was only talking in the scope of going to PR.
As an implementer, I agree that WPT is not enough.

Looking at WPT, I am for instance worried about some results that apparently are not passing in any browser, for instance tests related to RTCSctpTransport.
That could raise the question of when/if support is planned, whether we should make it an optional construct, punt the construct...