[webrtc-pc] setCodecPreferences vs unidirectional codecs (#2888)

fippo has just created a new issue for https://github.com/w3c/webrtc-pc:

== setCodecPreferences vs unidirectional codecs ==
(for after TPAC, this is a minor detail and we have more important things to discuss)

https://w3c.github.io/webrtc-pc/#dom-rtcrtptransceiver-setcodecpreferences
Step 6 says that
> If the intersection between codecs and RTCRtpSender.getCapabilities(kind).codecs or the intersection between codecs and 
> RTCRtpReceiver.getCapabilities(kind).codecs only contains RTX, RED or FEC codecs or is an empty set, throw InvalidModificationError. 
> This ensures that we always have something to offer, regardless of transceiver.direction.

This means that you must negotiate at least one codec that you can send and receive.

While this seems harmless, it causes trouble with asymmetric codec support and that is a fairly common thing. E.g. in Chrome on Windows we have  14 send codecs and 21 receive codecs, with the asymmetric ones ranging from H264 over VP9 to AV1.
The worst example we found was Android where the H264 send and receive profiles are not an exact string match and we have support for sending 42e032, 4d0032 and 640032 while being able to receive 42001f, 4d001f and 64001f. This lead to errors in a fairly innocent attempt to restrict the codecs to H264 variants:
```
const codecs = RTCRtpSender.getCapabilities('codecs')
  .filter(c => c.mimeType.toLowerCase() === 'video/H264');
someTransceiver.setCodecPreferences(codecs);
```

The sentence "this ensures that we always have something to offer" is there to avoid the subsequent problem, not being able to negotiate a codec when the transceiver direction changes like this:
```
t.direction = 'recvonly'
t.setCodecPreferences({a-codec-you-can-only-encode})
```
If this was taken to createOffer and setLocalDescription it might lead to an m= line that is rejected which would cause the transceiver to be  stopped which would be quite surprising.

setCodecPreferences might check the direction in that case but that just leads to the following scenario:
```
t.setCodecPreferences({a-codec-you-can-only-encode})
t.direction = 'recvonly'
```
One approach to solving this is that an attempt to set the direction to 'recvonly' might throw. This also requires a compability check in setCodecPreferences in case the direction was set before setCodecPreferences.

The other alternative would be to let createOffer (which knows about send+recv codes as well as the direction) throw an error when it detects this mismatch. It gets a bit finicky to figure out which transceiver was causing this, the mid *might* not be available.

Overall I think this is an edge case but the first example is a fairly common flow that is currently broken (depending on the codec support)

Also: how is the "intersection of codecs" defined? It seems that for H264 this must take into account whether level-asymmetry is allowed, no?

Please view or discuss this issue at https://github.com/w3c/webrtc-pc/issues/2888 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 11 August 2023 08:24:57 UTC