Re: [webrtc-encoded-transform] Need APIs for using WebCodec with PeerConnection (#121)

I think this API would be relevant to the E2E encryption use case as well as other use cases (e.g. AR/VR). 

The trickiest part will be the interaction of WebCodecs encoder rate control and WebRTC congestion control. While WebCodecs has an average bitrate target (which it can undershoot or overshoot in the short-term), it also has the ability to support SVC and/or simulcast, which allows the sender rate to be adjusted quickly (e.g. by dropping or adding layers). 

For this to be used effectively, the application needs to know when the potential sending rate drops (due to loss or increased delay)  as well as when the potential sending rate increases to the point where it might be possible to add back a simulcast or SVC layer that was previously dropped.  WebRTC's RTP transport is unique in supporting the latter scenario; transports such as `RTCDataChannel` or `WebTransport` do not support "probing" so as to allow faster rampup; they intrinsically optimize more for quality (e.g. a video upload scenario) than latency (video conferencing). 

-- 
GitHub Notification of comment by aboba
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/121#issuecomment-933934015 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 4 October 2021 23:32:17 UTC