[mst-content-hint] Bring Your Own Adaptation with "maintain-framerate-and-resolution" (#62)

henbos has just created a new issue for https://github.com/w3c/mst-content-hint:

== Bring Your Own Adaptation with "maintain-framerate-and-resolution" ==
Today, `RTCDegradationPreference` API solves two problems that sound similar but are in fact different:
1. It prevents overloading the encoder and the network. These are "hard constraints".
2. It tries to optimize for quality and performance under network constraints. These are "softer constraints" which necessarily involve tradeoffs that the app may care about.

Let's clarify the difference with an example. If I try to send 720p video with `maintain-framerate` and my BW is limited, the user agent has no choice but to reduce the target bitrate of the encoder, this is 1). Sending 720p video with a low target bitrate is possible, but the resulting video quality is suboptimal, so the user agent reduces the resolution until the quality looks OK (e.g. by regularly monitoring QP), resulting in sending 360p video instead, this is 2).

While the user agent should always be responsible for 1), the app may want to control 2). Possible use cases:
- Remote endpoint-aware choices: If the majority of the meeting wants HD, send "QVGA, HD" rather than "QVGA, VGA" when BW limited.
- If performance constrained, switch to HW encoder by reconfiguring scalabilityMode/codec.
- Dynamic preferences: Switch which aspect to degrade depending on content being shared (e.g. static or dynamic screenshare).
- Balanced 2.0: Degrade resolution at 4K but framerate once we've adapted down to HD rather than a little of both all the way down.
- etc...

Both "quality adaptation" and the poorly named "cpu adaptation" can be implemented from JavaScript by polling getStats and looking at QP and encode time. The problem is that if both the app and the WebRTC stack are implementing 2) at the same time we end up with race conditions, flip-flopping, the app never knowing what it's going to get and risk of compat issues if apps start depending on "how aggressive" a user agent's adaptation algorithm is. WebRTC's "soft adaptation" also has resets itself when you reconfigure which could have unintended consequences.

### Proposal

Introduce `maintain-framerate-and-resolution` which has the user agent provide 1) but not 2). The app may now poll getStats and perform setParameters without interference from WebRTC. If it doesn't, quality will be suboptimal but we will not overload encoder or network due to 1), i.e. user agent can always drop frames should the app not act on time.

Please view or discuss this issue at https://github.com/w3c/mst-content-hint/issues/62 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 3 September 2025 11:09:34 UTC