Re: [webrtc-encoded-transform] Is frame-level fanout in scope of WebRTC encoded transform? (#211)

My personal thoughts:
- On sender side, the encoded transform model is to transform encoded content just before packetization. It is constrained in a way so that the encoded transform works at the pace of the encoder.
- With frame fanout, there is no UA encoder, the web application is responsible to provide both the source data and encoder directly as encoded content. In that context, the pace of the encoder is no longer in control of the UA. This is an important change and it is appropriate for the UA to know this through declarative web APIs instead of heuristics.
- It also requires the UA to know what to do in case the sender is going over budget. Should it drop frames? Should it do buffering? Should it instruct the web applications to somehow reduce the throughput? This probably warrants an error/adaptation mechanism that WebRTC encoded transform does not require.

Given this, I would think a separate API is a better choice. It would also allows to use both APIs jointly.
I do not think the API surface would be very big and piggy backing on WebRTC encoded transform probably does not save us much here. Here is a quick illustration of what it could be:

```
[Exposed=DedicatedWorker]
interface RTCRtpSenderEncodedSource {
    constructor()
    readonly attribute RTCRtpSenderEncodedSourceHandle handle;
    // Need congestion API, error API and enqueue API
    undefined enqueue((EncodedAudioChunk or EncodedVideoChunk) chunk, DOMString mimeType)
};

typedef (EncodedAudioChunk or EncodedVideoChunk) RTCEncodedMediaChunkData;
dictionary RTCEncodedChunk {
    RTCEncodedChunkData data;
    DOMString mimeType;
}

[Exposed=(DedicatedWorker, Window),
Transferable]
interface RTCRtpSenderEncodedSourceHandle {
}

partial interface RTCRtpSender {
    undefined replaceTrack(RTCRtpSenderEncodedSourceHandle handle);
    readonly attribute RTCRtpSenderEncodedSourceHandle encodedSourceHandle;
}
```

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/211#issuecomment-1777458119 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 24 October 2023 15:15:57 UTC