Re: [webrtc-encoded-transform] Remove restriction on streams being limited to only one PC (#201)

> if people desire to go outside those restrictions, they use a different means of instantiating the input and output streams of frames that are the essential parts of this API for those purposes.

I thought about this approach a while ago. It is cleaner in the sense that it clearly states to the UA that (focusing on sender side) the JS application is responsible to implement the source+encoder part (which late fanout is clearly about). As an example, it will help UA provide meaningful WebRTC stats. Setting a track on such a sender could throw...

Exposing such API requires us to refine the WebRTC media pipeline model as we would expose things that are fully internal right now. This might not be unrelated to the TPAC Media/WebRTC WG discussion meeting I missed.

Related to this approach, my first questions would be:
- The frame level vs. packet level question. It would be great to see whether we can get consensus within the WG. For instance, if we have a packet level API, do we also need a frame level API? Do we need both?
- @jan-ivar's question: should fanout be done in JS or done by UA with web app tuning it via knobs?

To refine the WebRTC media pipeline model, there are a number of questions that might be useful to tackle, for instance:
- the bandwidth allocation model: WebRTC engine is providing a bitrate that the encoder needs to respect. What if it is undershooting or overshooting it?
- the timing model: WebRTC engine is processing frames in a given time-based order. What if JS encoder is sending frames differently?
- Get consensus on https://github.com/w3c/webrtc-encoded-transform/issues/109.
- the feedback model: WebRTC engine is providing instructions on the encoder (I need a key frame) based on RTCP feedback. Should we expose this?
- the error model: WebRTC engine can starve the encoder with input. What if JS is sending frames when UA does not expect it?
- the overuse model: WebRTC engine is preventing excessive CPU use by decreasing frame rate and/or resolution. Should we expose this? Probably not but worth digging.
- the threading model: WebRTC engine is probably currently doing this processing in background threads. Should we put the same restriction on the web?

Hope this helps moving forward.

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/pull/201#issuecomment-1740456199 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 29 September 2023 07:44:54 UTC