Re: On the way to WebRTC 1.0

On Fri, Oct 19, 2018 at 4:20 AM youenn fablet <yfablet@apple.com> wrote:

> Hi,
>
> I thought a bit on how to best finalize WebRTC 1.0 in terms of
> specification work.
> I believe we need to at least do two things:
> 1. Resolve all WebRTC 1.0 issues
> 2. Prove (to ourselves and others) that the WebRTC 1.0 spec is
> implementable in an interoperable manner
>

sounds reasonable.

>
> For 1, it might be a healthy exercise for the Working Group to label
> issues as "WebRTC 1.0” vs. “Post WebRTC 1.0”.
>

sounds good to me.


> [..]
>
> For 2, analysis needs to be done on the whole spec to identify holes in
> our testing coverage.
> I think this effort was done in the past (DrAlex?), maybe there is a need
> for refreshing it once again?
>

We do it at every meeting since the Lisbon meeting in 2016. This year again
Soares, who wrote 1000+ of the current 1300+ webrtc tests, will report on
the states of WPT. I will report separately on webrtc interoperability.

Note that WPT calls interoperability the capacity for the same API to
behave according to the spec ACROSS browsers (one browser at a time),
while in webrtc interoperability means that two browsers can establish a
peerconnection (and some more).It means unified pan vs plan B, it means
JSEP, codecs, simulcast, .... We try to raise the potential ambiguity in
the past, with no result. We should be very explicit in the scope of webrtc
to manage expectations.

Since the spec will continue evolving, we might also want to take some
> explicit "test step" for any PR being made so that we can classify each
> change as:
> - Editorial/No need for test
> - Need for WPT test (and a link to the corresponding WPT test PR)
> - Need for manual test/investigation or KITE test
>

While our feeling is that it would be a good thing indeed, our experience
with WPT is that a happy few do whatever they want. Review is broken, there
is no governance. There is no process in place to enforce that the approach
you propose, and it would be likely ignored if it goes in the way of the
happy few, or is perceive by them as without merit.


>
> At the end of the day, when we need to prove interoperability, we should
> be able to show:
> - A WPT report showing that each WPT test is passing in at least two
> browsers (or we have explanations on particular failures).
> - A manual report and/or KITE test report that shows interoperability
> between two browsers.
> Given that we are targeting the WebRTC 1.0 spec, I would believe that WPT
> should be our topmost priority.
>

Agreed.


>
> The current WPT report (https://wpt.fyi/results/webrtc?label=experimental)
> shows that we have some work in that area to either improve the tests, the
> implementations or the report.
> Some reasons why the report is so red:
> - Chrome is Plan B by default
> - Safari does not expose host ICE candidates.
> Reports gathered directly from browser CI might prove to be more green
> than from wpt.fyi.
>

There will be more granular results in our slide, but the webdriver side of
things is not helping either. manual testing might provide better result as
wpt.fyi is done using webdriver, on a limited (in terms of configuration),
non webrt-aware grid.


>
> Hope that helps,
> Y
>


-- 
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard

   -