Re: On the way to CR for webrtc-pc

> I'm not super familiar with what transitioning to CR entails, but can you
> say something about what goals you have for interoperability, in both the
> "implementations pass the same tests" and "two implementations can
> communicate" sense) sense?
>
>
Hi philip,

Dom, harald and myself have been working on the test suite for media stream
and webrtc(-pc). I have been so far in charge of reporting to the WG during
meetings, and especially TPAC. MediaStream received more love than webrtc
so far. After discussion with dom, harald, stefan and dan burnett, among
others, the consensus was that there is no tool to compute a "coverage" of
the spec, so for TPAC in lisbon last year I computed a "manual" coverage by
counting all the formative lines in the specs, and checking from the test
which were tested. While media stream was above 50% (still not a feat),
webrtc was below 25%.  The granularity is the line, which is less than
ideal, but it was a good start, and we had absolutely no visibility before
that.

Some discussions took place about parts of the specs (the algorithms
description) being normative or informative and other details. Discussion
about which version should be the reference (editor draft vs latest draft)
also took place.

Some discussions took place about parts of the specs not being testable
without "interoperability" testing, and I presented a possible framework
for it.

Some discussions took place about the problem of testing (even manually)
parts of the specs that require interaction with the security prompt. While
automating the interaction with the security prompt is a challenge in
itself, workaround have been implemented by at least by FF and Cr using
command line arguments, and profiles. However, it supposes the setting
remains the same during the entire test, while testing the spec would
require, e.g. to cancel an already provided approval to hardware access
before the promise return with the corresponding media stream. Fake sources
can also be provided the same way. The safari way, which allow for
programatic modification of the settings in the tests, and could test those
cases, was presented.

Some discussion took place about having both FF and Cr using libwebrtc
under the hood. Was it different enough to uphold the 2 browsers rule. The
answer from Dom / W3C was that, as far as W3c was concerned, the
implementations were different enough (network stack, ice stack, ...).




> In my work to bring Blink's Web IDL files closer into alignment with the
> specs that we link to, I'm mostly looking for non-standard things that need
> attention, but I also notice things that are in specs but not in Blink.
>
> On RTCPeerConnection, there's at least currentLocalDescription,
> pendingLocalDescription, currentRemoteDescription, pendingRemoteDescription
> and canTrickleIceCandidates. There are also things from partial interfaces
> that are harder to spot, like the sctp attribute. Some of these are not in
> Gecko either.
>
> Would it be helpful to the WG with a list of things that seem to
> implemented in <2 browsers, or what criteria are you using?
>
> Regarding test coverage, https://github.com/w3c/web-
> platform-tests/tree/master/webrtc is fairly limited and improving
> automated testing seems blocked on things like https://github.com/w3c/
> web-platform-tests/issues/5563. Automation aside, how will you determine
> the coverage, i.e. how do you make sure that the sctp attribute is tested
> and that a lack of implementations would show up as test failures?
>

So far, manually :-(  I'm open to suggestions!

-- 
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard

   -