Re: On the way to WebRTC 1.0

Youenn said:


I thought a bit on how to best finalize WebRTC 1.0 in terms of specification work.
I believe we need to at least do two things:
1. Resolve all WebRTC 1.0 issues

[BA] At TPAC, we have allocated time for discussion of  roughly a baker's dozen open issues on webrtc-pc. The breakdown of issues (45) by label is as follows:

For discussion at TPAC:                          13
Editorial:                                                   8
Question (Issue not yet clear):                 4
Needs submitter/assignee action            4
PR exists                                                  3
Enhancement (e.g. not WebRTC 1.0):       3
Simulcast  non-TPAC                               2
Pending RTCWEB WG Action:                 1
Miscellaneous                                         7

2. Prove (to ourselves and others) that the WebRTC 1.0 spec is implementable in an interoperable manner

[BA] At TPAC we will be having discussions relating to conformance testing (WPT) as well as protocol interoperability testing (KITE).


For 1, it might be a healthy exercise for the Working Group to label issues as "WebRTC 1.0” vs. “Post WebRTC 1.0”.

[BA] The "Enhancement" label is applied to Issues that will not be handled within the WebRTC-PC specification.


Since the spec will continue evolving, we might also want to take some explicit "test step" for any PR being made so that we can classify each change as:
- Editorial/No need for test
- Need for WPT test (and a link to the corresponding WPT test PR)
- Need for manual test/investigation or KITE test

[BA] We have had a "test before commit" policy in place since TPAC 2017.  We will be going over how well this process is working at TPAC.  Overall, I would note that areas of the specification that are most problematic to implementers are have so far have been difficult to test with WPT  (e.g. simulcast). We will be talking about approaches we might use to address this.

At the end of the day, when we need to prove interoperability, we should be able to show:
- A WPT report showing that each WPT test is passing in at least two browsers (or we have explanations on particular failures).

[BA] WPT tests API conformance, not protocol interoperability. Note that  in a number of cases, the results of the WPT tests do not reflect the true status because the tests have dependencies orthogonal to the functionality being tested that cause them to fail.  For example, there are tests that rely on getUserMedia which will fail if the prompt cannot automatically be dismissed, even if they would pass if a Track could be provided by other means (e.g. generated by WebAudio).  So things are not quite as bad as the WPT test matrix would make them out to be. The confluence matrix (which shows which APIs are implemented) appears more "green", for example.

[Youenn] Given that we are targeting the WebRTC 1.0 spec, I would believe that WPT should be our topmost priority.

[BA] I would agree that testing overall is very important (not just WPT but also KITE).  I am looking forward to a discussion of how we might improve the process and raise the level of engagement so as to make better progress.

  *