Re: [webrtc-extensions] Add API to control encode complexity (#191)

> QP settings like [MinAllowedFrameQP](https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_minallowedframeqp) and [MaxAllowedFrameQP](https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_maxallowedframeqp).

These settings will have an impact on bitrate, I am not sure it will have any effect on CPU.

> [PrioritizeEncodingSpeedOverQuality setting](https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_prioritizeencodingspeedoverquality)

I am not sure this one kicks in in the low latency code path.
Overall, when hardware encoders are in use, CPU usage will probably not vary much.
I do not know how is mapped KEY_COMPLEXITY to HW realtime android encoders and what the impact is.
The OP is mentioning SW encoders, is this the target?

> the application detects a bad network quality and that resources are available

I am not exactly sure to understand how the application is supposed to know that resources are available and I am questioning how we can get interop between UAs.
For instance, CPU adaptation could be done either with the encoder complexity mode or with media adaptation, ditto for bitrate.

The following might somehow work for video:
- web site sets encoding complexity, UA translates it to a fixed value.
- When UA hits bitrate or CPU contention, UA is not expected to change video encoder's complexity mode. Instead it relies on the current degradation preference by either reducing frame rate or resolution.
- web site may further tweak encoding complexity based on observed sent frame rate / resolution / bitrate.

Is this how this is supposed to be implemented and used?

With regards to audio, it cannot really do media adaptation, I am unclear how it would know CPU adaptation is needed.

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/webrtc-extensions/issues/191#issuecomment-1898046504 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 18 January 2024 08:51:14 UTC