Re: [webrtc-encoded-transform] Add description of an API for controlling SDP codec negotiation (#186)

> - Plug a new codec. It seems best to solve this with a separate API, I do not think we want to enshrine this kind of support in encoded transforms. For instance, a video encoder takes a VideoFrame as input and EncodedVideoChunk as output. A script transform takes an EncodedVideoChunk both for input and output.

I agree this suggests an API mismatch. I see evidence of this in the [Lyra use case](https://www.meetecho.com/blog/playing-with-lyra/) as well, which had to:
 1. Use a special "L16" null-codec in Chrome to bypass encoding and pass raw audio directly to the transform
 2. SDP-munge `ptime` back to 20, to counter Chrome's drop to 10 ms from the large amount of (uncompressed) input
 3. Reverse the 16-bit [endian](https://en.wikipedia.org/wiki/Endianness) of the input, which is already in network order, back to platform order for lyra-js

It's an impressive feat, but I imagine these problems would be amplified with video. It might need its own API.

> Also, these two should be usable together: it makes sense to use a SFrame transform on a plugged-in codec simply with `sender.transform = sframeTransform`.

This makes sense to me as well. E.g. spitballing:
```js
sender.encoder = new RTCRtpScriptEncoder(worker, options);
sender.transform = new SFrameTransform({role: "encrypt"});
```
?

-- 
GitHub Notification of comment by jan-ivar
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/pull/186#issuecomment-1765083234 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 16 October 2023 18:46:10 UTC