[webrtc-extensions] RTCRtpEncodingParameters: scaleResolutionTo (#159)

henbos has just created a new issue for https://github.com/w3c/webrtc-extensions:

== RTCRtpEncodingParameters: scaleResolutionTo ==
We all know and love `scaleResolutionDownBy`... it let's you do this:

Capture 720p and apply expensive video effects on the track.
Send {active, scaleResolutionDownBy:1 (720p)} + {active, scaleResolutionDownBy:2 (360p)} simulcast.

But then server tells us 720p is not needed, so we send {inactive, active:360p}.
Now you might wonder, why are we applying expensive video effects on a 720p track if we're only sending it in 360p?

So we track.applyConstraints() to 1/2 downscale the track and we setParameters() to 2x the scaling factors to counter the fact that the input frame size is smaller. The end result is:

360p track sent as {inactive} + {active, scaleResolutionDownBy:1 (still 360p)}.

**Problem:** This is racy. The track changing size and the parameters updating the scaling factors are not in-sync, so you might send 720p on the VGA layers for a few frames. You're wasting a key frame on sending the wrong resolution and then you're wasting another key frame to adjust back to the desired 360p resolution. Working around this by temporarily inactivating all encodings while the track is resizing will appear like glitches to the receiver of the stream and there probably would still have to be key frames when the encodings are re-activated. The API was not designed for this.

**Proposal:** Add "scaleResolutionTo: 360" API which is equivalent to always having the correct scaling factor. In the above example, changing the track's size from 720p to 360p would not result in reconfiguring the encoder and there would be no key frames or races.

Please view or discuss this issue at https://github.com/w3c/webrtc-extensions/issues/159 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 13 April 2023 08:23:46 UTC