Draft minutes of the WEBRTC WG Virtual Interim (May 25, 2021)

#WebRTC Interim May 25, 2021

Notetaker: Tim Panton


##Capture Handle - Elad Alon


Slide 6: Scenario: Video conferencing capturing a slideshare application.

Conferencing application exposes a "Share" button that calls getDisplayMedia.

User then chooses an application to share.


Slide 7: Slideshare app supports an API for things like "Next Slide".

Conferencing app needs to discover:

  * The captured application’s identity.

  * The session ID.


Slide 8: Without changes to the browser, solutions are hacky.

The shared application could embed a QRcode that could be read by the conferencing app.

This is inefficient and prone to error.


Slide 9: Suggested solution

Captured app:

    Allow opt-in exposure of origin.

    Allow a custom "handle" to be set

    Alow captured app to specify which origins may read values in the "handle"


Capturing application

    Allow checking the current capture-handle

    Allow registering a handler for capture-handle-update


Slides 10 & 11: Code example


Slide 12: Origin might not be needed if small known set of servers


Slide 13: WebIDL


Slide 14: Privacy/Security concerns

Appears to not entail new risks above those of the QR code alternative


Discussion:

Youenn: Prefers to always expose origin.

Elad: will consider

Youenn: One time event?

Elad: no - captured page might navigate.

Tim: is this always server based collab?

Elad: No, same origin can collaborate within the browser

Jan-Ivar: relation to getViewport media + limiting size and rate of data passing.

Elad: competition is healthy, this is useful now, not good to delay.


##Jan-Ivar getViewportMedia() in 2023.

Slide 15: All major presentation web sites use site isolation and opt into html capture.

They register for preferential treatment in getDisplayMedia's new picker

Elad: Doesn't this create an icentive for apps to lie to get to the top of the picker?

Jan-Ivar: What is the picker order in the trial?

Elad: Most recently interaction from user

Youenn: If this is a message channel why don’t we use the existing channel mechanism ?


Slide 16: In Origin Trial in Chrome M92

Intent to Prototype notice: https://groups.google.com/a/chromium.org/g/blink-dev/c/yLTykllpNmI


Bernard: We're out of time.  Next steps ? Summary?


*Jan-Ivar: agree there is a problem to be solved, but no consensus on the proposal.*

*Bernard: Next step is discussion on the mailing list and github.*


## Elad Conditional Audio Suppression PR164

Slide 17: Assume we have multiple people taking turns presenting to a projector and set of speakers.

When next speaker presents a tab, do they want the audio to flow out of both their laptop’s speakers as well as the PA system’s? Probably not.


Solution? Define a new constraint - suppressLocalAudioPlayback.

When this constraint is applied, audio from captured source is not played out over the local speakers.

A captured page should be able to have audio muted


Elad + Jan-Ivar: Discussion of role of application vs the user agent.

Elad : Not suitable for picker.

Elad: User needs to be made aware and possible to reverse decisions


Bernard? Next steps ? Summary?

*Discussion moves to Github…*


##Elad : Region capture

Slide 19: Assume we have getViewportMedia and apps can now self-capture.

How does an app crop its video stream.  Why would it want to?


Slide 20: Use case: hiding speaker notes.


Slide 21: Performance and security needs means it must be done in the browser not the application

Slide 22: How to specify the capture region? Use an ID


element != region because of occlusion.


Slide 25: API shape slide


Jan-Ivar: Better to tie to iframe not an element


Bernard: Conclusions?  Next Steps?

*Youenn: There is a consensus that cropping is desirable.*

Discussion about what is posted - id or a track?

API surface / crop rule is still for discussion.


##Youenn SFrame error handling Issue58/PR103


What is the way for the native transport to report errors in E2E encryption/authentication?

Bernard : Does this affect NACK?

Youenn: No, this is only about errors in E2E; missing sequence #s/recovery and SRTP encrypt/decrypt/auth are not affected..

Likely usage would be issues with key synchronization (e.g. keyID changes, so JS needs new key) or attack by rogue SFU.

Jan-Ivar: are errors ever fatal? Youenn: no

Bernard: Rekey timing is critical

Youenn: skey spec should avoid this.


Bernard: Conclusions?

*Harald: rough consensus*


##Youenn: use  webcodec Encoded Chunk Issue 99

Slide 28: WebCodecs defines EncodedAudio/VideoChunk, which is immutable.

Encoded transform has RTCEncodedAudio/VideoFrame which is not immutable (data ownership transferred at write step).

Other difference is that RTCEncoded*Frame has RTC-specific metadata added.


Options: keep separate APIs (plus Duck taping) or Merge APIs


Harald: it was a mistake that they are different

Bernard: EncodedChunk is immutable to prevent an application from writing into the encoded chunk at the same time the

WebCodecs encoder was producing it.

So if you want to encrypt encoded media, you need to copy the chunk into an ArrayBuffer and encrypt it there.

Youenn: Couldn't it have been handled by locking instead? That could save a copy.


Discussion to be continued…. (Raise issue of locking in webcodecs)


##Harald - Mediacapture-transform issues

Slide 30: 3 issues under discussion


Slide 31: Readable/writable Stream right approach ?

Harald: I think Streams is the right approach. Developers like it, no significant downsides. No alternatives have been proposed.



Youenn: Consensus is still evolving on issue  but not there yet. Will propose alternative next month.


Conclusion: no consensus on this issue yet, will be covered at next interim.


Slide 32: Two major alternatives:

1. Worker processing easy and main thread possible

2. Worker processing mandatory and main thread hard.


So far, all arguments have come from browser vendors, not developers.


Bernard: We have some developers at this meeting, including one (Matt Gordon) working on a low latency application

that currently does transport, decode and rendering all on the main thread.

Tim Panton: default to worker especially for new users.

Guido Urdaneta: One developer had a prototype where using workers produces complexity for transpiled apps.

Possible to tag objects so that by default can only run in a worker - but can be tagged to run on main thread

Matt Gordon: My experience is that Garbage Collection on the main thread can wreak havoc. So using threads is

highly desirable. However, keep in mind that APIs such as RTCDataChannel and MSE do not currently run in

dedicated workers. So there are applications running on the main thread today due to that, and it is easier

to migrate in stages (e.g. move from MSE to WebCodecs + mediacapture-transform, then move to Dedicated Workers).


Harald: don’t want to make it so hard to run in main thread that people give up

Youenn: Prototype in worker only then perhaps add shim to support main.

Jan-Ivar: Main thread is over used and under appreciated


Harald - relates to next issue.

Conclusion: no consensus on this issue yet.


##Harald: Issue 16 Transfer MediaStreamTrack

Slide 33: comment 21 Transferable (track is moved) (life is tied to original doc)

vs

comment 24 Serialzable (track is copied) (life is tied to source)

Guido: They aren't exclusive.  It's possible to support both.


Jan-Ivar : Prefer Move but make possible to pass track back to main thread.

Transferable (move) preferred because of leaky unnecessary clones in Serializable

Youenn : Also transfer between iframe.


Bernard: Summary/Next steps?

*Consensus for Transferable*


Youenn: complexity is in the lifetime management

Agree to file a new issue on lifetime management.


Slide 34: We finished the slides! (Picture of an Iguana)

Jan-Ivar: Have we run out of birds?

Bernard: Haven't been bird watching since the pandemic. So we're stuck with lizards.