Minutes of the April 27, 2021 WEBRTC WG Meeting

Minutes of the WEBRTC WG Virtual Interim
April 27, 2021

Chairs: Bernard Aboba, Harald Alvestrand, Jan-Ivar Bruaroey

Scribe: Nick Steele

Agenda: https://www.w3.org/2011/04/webrtc/wiki/April_27_2021<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.w3.org%2F2011%2F04%2Fwebrtc%2Fwiki%2FApril_27_2021&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459173635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Lb8%2BUc1sEz4bUYJh38RevrFlCY3DNgvZQhzbjenFLSo%3D&reserved=0>

1. Dr. Alex Gouaillard (Sergio Garcio Murillo)

Dr. Alex Gouaillard passed away Thursday, April 8th, 2021.

Bernard: Dr. Alex performed cutting edge work that didn’t always get a lot of recognition but he never shied away from it.
Sergio: It will be hard to regroup without him but we hope to continue to work on the cutting edge.
Time Panton: Dr. Alex helped with Safari and Internet Explorer adopting WebRTC.
Youenn and others: Dr. Alex was extremely gracious, always warmly received new people, was always willing to help others, and will be missed greatly.

2. WebRTC-PC extension process (Harald Alvestrand)

Proposal: WebRTC scratch pad for new proposals. Specs can incubate there. Once we’ve looked at them and kicked the tires we can merge them to the main doc. The exception to this is fixing bugs if they exist in the spec.
We should aim to merge extension docs into the spec within 6 months.

The proposal was seconded by Henrik, Youenn.

Youenn: Suggested three steps: 1) Proposal. 2) Fixes. 3) Final stages. He will make a comment in the PR describing the process in further detail.
Harald: Cautioned that we shouldn’t assume we have consensus at the outset.
Jan-Ivar: This is a place where all ideas can be added, whether they are good or not.
Henrik: We could do the opposite of “implementation at risk”, i.e. by default they are at risk.
Tim Panton: Concerned what web developers see: It should be reasonably clear what browsers do.
Jan-Ivar: Sees no issues with discussions in private repos with things that need more bake time.

AI: Youenn to make a comment in the PR describing his proposed process in further detail.
AI: Please make more comments on PR #2637 (https://github.com/w3c/webrtc-pc/pull/2637<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwebrtc-pc%2Fpull%2F2637&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459173635%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Zcina8LKJAGyjFoUwjKsL8O4yp6vt8iihTVvkkPmJBo%3D&reserved=0>) (this conversation was shelved in the interest of time)


3. Testing (ongoing from previous two meetings)

There is an issue with the crc32 library.

AI: Will move forward on RFC and in parallel try to address the licensing issue.


4. requestKeyFrame API (Bernard Aboba)

This is actually about generating keyframes and is a misnomer. This was put into the icebox. This is not reliable on all browsers, and may be due to a race condition. The spec does not say if you should generate a keyframe.

Suggested (Bernard): A new API called "generateKeyGrame"

Use cases:

a. When people join or leave a conference.
b. When servers switch between simulcast recordings.
c. End to end encryption with iframes.

AI: We are going to update PR37 (https://github.com/w3c/webrtc-extensions/pull/37<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwebrtc-extensions%2Fpull%2F37&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459183593%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Lh2NbKUR2VOpgTwiooUhGbqxdeZCnvhA%2FtKu9uyhecY%3D&reserved=0>) to go in this direction, with permission. Comment on the PR and get it closer to being finished.


5. MediaCapture Transform: proposal to adopt (Guido Urdaneta)

Implemented in Chrome/Edge (M90+). 50+ signups, 8+ are experimenting with Breakout Box. Positive feedback in general. Apps in active development by Zoom & Google.

Issue 4 (https://github.com/w3c/mediacapture-transform/issues/4<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fmediacapture-transform%2Fissues%2F4&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459183593%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XP7Xm6kToEp6Dl2xbzG23usKvlM6Ha9p5xhphKy%2BkkQ%3D&reserved=0>)

We believe using Readable/WritableStream is the right approach. It allows direct support for workers, it’s familiar, and well known. Separating them allows more use cases:

a. Custom sinks (Zoom is testing this).
b. Custom sources.
c. Multiple-source tracks.

Issue 6 (https://github.com/w3c/mediacapture-transform/issues/6<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fmediacapture-transform%2Fissues%2F6&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459183593%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OaMg0gZXaHMoGXFZCBXg17w0MUZiKX7J299CV%2BYbCWc%3D&reserved=0>)

What if the upstream track produces more frames than can be consumed? Two existing mechanisms mentioned: Application and UA level.

Issue 20 (https://github.com/w3c/mediacapture-transform/issues/20<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fmediacapture-transform%2Fissues%2F20&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459193552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7LvWm%2BTzV8w8jvtVR1XCaq8K7dASAYJf%2Bl7zWlqoMjM%3D&reserved=0>):

Add “real-time” warning/note to MediaStreamTrackGenerator.
What if this generates more frames than the sinks can consume? How do we notify the application?

a. Error-reporting mechanisms
b. Define a signal exposed in MediaStreamTrackGenerator.readableControl

Issue 1 (https://github.com/w3c/mediacapture-transform/issues/1<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fmediacapture-transform%2Fissues%2F1&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459193552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vqauL5UzT%2F2j7hUzVKWKpo5h9ckEz1EmCYL4AqycKPA%3D&reserved=0>)

Add a high-quality face/body tracking API to discourage poor/discriminatory implementations.

6. Proposal: Adopt the spec as a WG (Working Group) document.

That means:

a. WG agrees that the problem needs to be solved
b. Further changes will be done by WG
c. WG has the right/duty to make changes to the API if in interest of the Web community.

Bernard: It was confusing to move to Breakout Box without documentation and may be why only 8 signups to Breakout Box.
Jan-Ivar: The repo is only being contributed to by a single company. Chrome is planning to ship this even though the document has not been adopted by the WG and is concerning from a process standpoint. If "Working Draft" is the point when things will be shipped on the web, we should pay more attention to when things become "Working Drafts". For example, WebCodecs had a video track reader, but not audio track readers, so Breakout box now processes audio tracks. We should not be shipping before horizontal review, and horizontal review is not taking place at this point.
Yournn: This happened last year; Chrome shipped a proposal, but we decided to change the API afterwards. Wants to make sure this does not happen again.
Chris Cunningham: WebCodecs supports audio/video in Breakout Box.

7. Feedback on Transform (Youenn Fablet)

For video we need a better solution that Canvas. For audio we have Audio Worklet which is fine. Zoom is using it for mic and remote audio rendering and it is working fine for them.
Harald: Audio Worklet is synchronous (runs immediately), and therefore doesn’t work for all use cases.
Chris Cunningham: Received feedback from developers along the same comments by Harald; you can drop audio, but this issue isn’t present with Breakout Box. This is also a lot of hoops to jump through if you’re only planning to do something other than render the audio.
Paul Adenot: There are implementation problems since shipping with performance in Chrome (not an issue in other browsers).
Youenn: Audio worklet has a higher priority thread, is more resilient (no sync XHR). We can discuss audio worklet extensions because it does not have control signals, i.e. it can be extended or shimmed. If you check MaxBufferSize in the main thread, you get a bug. Since Chrome 91 media does not touch the main thread.
Youenn: Suggested documenting use cases on controlling channels before talking about the API.
Raw video media access API is a great idea, but with audio there is no consensus.
Youenn: Proposed focusing on video and core functionality in V1.
Jan-Ivar: Back-pressure doesn’t necessarily mean buffering. I hope we can all keep this in mind. Example on Slide 29.

Consensus could not be achieved.

AI: Youenn will provide some deeper examples on issues mentioned to attempt to achieve greater consensus.
AI: We will transfer this discussion to the list in the interest of time.

8. Remove redundant API (Youenn Fablet)

PR 64: https://github.com/w3c/webrtc-encoded-transform/pull/64<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwebrtc-encoded-transform%2Fpull%2F64&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459193552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=coeJkEhWn9fYo91rzeLZI9SVAgkQB87kU%2BPkTf57aqE%3D&reserved=0>
Youenn: We should move to a transform model. Remove createEncodedStreams from the specification.

None opposed. Consensus achieved.

AI: Drop the createEncodedStreams (redundant) API, and come up with use cases for main thread access. Reusing the API or changing the new one.

9. Issue 99: Use WebCodec APIs (Youenn Fablet)

https://github.com/w3c/webrtc-encoded-transform/issues/99<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwebrtc-encoded-transform%2Fissues%2F99&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459203503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GgrujRAiUvmWUxPP2ph7uaCA56%2BGK3ncURRki9GRP68%3D&reserved=0>

This issue was skipped in the interest of time.

10. Issue 155: getViewportMedia (Jan-Ivar Bruaroey)

https://github.com/w3c/mediacapture-screen-share/issues/155<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fmediacapture-screen-share%2Fissues%2F155&data=04%7C01%7CBernard.Aboba%40microsoft.com%7C0bae1591573c4d1d72fc08d90aa820a4%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637552540459203503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qPtyZ%2BPxmp6ylLW7b7gDECaMDmI%2BXoltp%2FwcheWgUhI%3D&reserved=0>

a. Only expose if window.crossOriginIsolated (a.k.a. COOP+COEP)
b. Resources are on their own (vulnerable to Spectre anyway) 👻
c. Failure mode: Block loading of non-opt-in iframes (generalizes well)

Agree on: Shape of opt-in-to-capture doc header.

Problems w/COEP: require-corp; html-capture. Opts into riskier (not safer) profile (confusing). Requires Fetch Metadata request header as well (Sec-Fetch-COEP)

Proposal: Consensus on all of the above (i.e., except for the proposal from @camillelamy).

No objections.

getViewportMedia: Mitigate User information harvesting

Jan-Ivar: This is still unclear.

getViewportMedia: Cropping

Capturing intersection of viewports which will capture the smallest possible surface, including everything on top of the iframe. Limiting capture to yourself.

Elad Alon: What is being captured can be confusing to developers.

AI: No objections to getViewportMedia.

Meeting was adjourned due to an expiration of the allotted time.