Re: [webrtc-encoded-transform] Remove restriction on streams being limited to only one PC (#201)

As discussed during yesterday's editor's call, here is some feedback for this issue.

This 1 pc restriction is similar to the restriction to preserve the ordering of chunks. It prevents potential footguns. It removes edge cases. It helps browser interoperability. It is consistent with the WebRTC media pipeline and encoded transform model. As it is currently, WebRTC encoded transform is nothing but an encoder post-processor or a decoder pre-processor. No encoder/decoder -> no processor.

For instance, if a website tries to write too much data, the UA will drop frames before encoder to enforce bandwidth allocation. The transform will not receive any new chunk until the sender gets back some allocated bandwidth slot. Another example is that the web app cannot send data through the sender transform of a recvonly transceiver. Ditto on receive side where track may be muted. These are good properties. 

Removing this restriction would change the model of WebRTC encoded transform, and more generally of the WebRTC media pipeline. We should favour proposals that can implement the use cases by staying consistent with the existing model we agreed on.

AIUI, the use cases can be implemented by new APIs, located elsewhere than WebRTC encoded transform. Two examples come to mind:
- A low level API that would allow building an SFU in a browser. This could be the RTCRtpTransport API presented at TPAC. My understanding from TPAC meeting is that there is interest in moving forward with this proposal.
- A high level API that would allow forwarding received media (without reencoding whenever possible) via another peer connection, as mentioned by @jan-ivar. Say for instance `sender.enableMediaForwarding({ ... })` or `sender.replaceTrack(track, { forwarding: true, ... });`. That API could be complemented by progressively adding knobs to control the exact forwarding behaviour, as we discover more needs from applications. This is inline with the current RTCPeerConnection model we all like.

Above proposals can lead to good interop and good user experience (minimum latency, minimum CPU overhead...).

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/pull/201#issuecomment-1740387694 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 29 September 2023 06:46:06 UTC