Re: [webrtc-svc] should scalabilityMode be an array scalabilityModes? (#50)

@fippo Let me try to explain how the negotiation works (or at least what might be involved).  Through discussions with Danil and Chris I've come to appreciate that it is somewhat more complex than I had initially thought. 

In a conferencing scenario (where SVC is most likely to be of use), the SFM will typically be the Offerer and the browser the Answerer.  In that situation, the SFM will Offer what codecs it can send or receive and the browser will Answer. 

To figure out what codecs/modes the browser can send to the SFM, the browser can compute the intersection of RTCRtpSender.getCapabilities("video") and the SFM's simulated RTCRtpReceiver.getCapabilities("video"). 
Note that for a mode to be receivable by the SFM, the browser and SFM need to jointly support an RTP header extension that makes this possible (e.g. the browser can't send "L2T2_KEY_SHIFT" without negotiating the AV1 Dependency Descriptor).  

The codecs/mode that the SFM can send to the browser is determined by the RTP header extensions that they jointly support, as well as by whether the browser supports `referenceScaling` for the hardware/software codec.  For example, an AV1 hardware decoder might not support `referenceScaling` in which case the browser won't be able to decode spatial scalability modes (but would still be able to decode temporal modes).  So in that case (where the browser required hardware acceleration to obtain the required performance) the SFM couldn't send spatial scalability to the browser even if DD was negotiated. 

For reference, Chromium currently supports the following scalabilityMode values for AV1: 
        "L1T2",
        "L1T3",
        "L2T1",
        "L2T1h",
        "L2T1_KEY",
        "L2T2",
        "L2T2_KEY",
        "L2T2_KEY_SHIFT",
        "L3T1",
        "L3T3",
        "L3T3_KEY",
        "S2T1"

For VP8/VP9 ("profile-id=0" only), the following values are advertised: 
        "L1T2",
        "L1T3"

Some implications of this: 

There are situations in which the codec used for sending may differ from the codec used for receiving (e.g. AV1 decoding is further along than AV1 encoding).

If the browser wants to send modes that are only supported in AV1 (e.g. "S2T1" or "L3T3_KEY") then it needs to prefer AV1 in the Offer and Answer. 


-- 
GitHub Notification of comment by aboba
Please view or discuss this issue at https://github.com/w3c/webrtc-svc/issues/50#issuecomment-989487984 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 9 December 2021 03:59:55 UTC