[webrtc-encoded-transform] How to support both sframe and JS transform at the same time (#266)

jan-ivar has just created a new issue for https://github.com/w3c/webrtc-encoded-transform:

== How to support both sframe and JS transform at the same time ==
Turning on sframe encryption alone is super easy, barely an inconvenience (using per-packet as an example):
```js
transceiver.sender.transform = new SFrameTransform({mode: "per-packet"});
```

Goodbye JS-transform (`RTCRtpScriptTransform`) for E2EE! Except what about its other use cases: adding metadata and fan-out?

<img src="https://github.com/user-attachments/assets/0ca7870d-0d6e-4004-8d80-1a850b39254e">

### Model A: sframe and JS transform as orthogonal features

With sframe handled entirely in the browser—near packetization and depacketization—a first principal approach might arrive at letting websites turn it on independently, leaving everything else the same, including `RTCRtpScriptTransform`. Pick a switch:
1. ```js
    transceiver.sender.transform = new RTCRtpScriptTransform(worker);
    transceiver.sender.stransform = new SFrameTransform;
    ```
2. ```js
    transceiver.sender.transform = new RTCRtpScriptTransform(worker, {sframe: "per-packet"});
    ```
3. ```js
    transceiver.sender.transform = new SFrameTransform(worker, {mode: "per-packet"});
    ```
4. ```js
    transceiver.sframe = "per-packet";
    ```

Ditto receiver. This switch would turn on sframe encrypt/decrypt at X (per-frame) or Y (per-packet):

- MST → encoder → JS transform (optional) → (X) packetizer (Y) → network 
- play ← decoder ← JS transform (optional) ← (X) depacketizer (Y) ← network

Works the same with both per-frame and per-packet, existing unchanged JS transforms, and without JS transform.

### Model B: sframe as a feature inside JS transform

A single line in the spec `SFrameTransform includes GenericTransformStream;` suggests SFrameTransform doubles as a Transform _Stream_ https://github.com/w3c/webrtc-encoded-transform/issues/262.

This suggests another way of combining the two features might have been intended:
```js
// main.html
transceiver.sender.transform = new RTCRtpScriptTransform(worker, {type: "sframe"});

// worker.js
onrtctransform = async ({transformer: {readable, writable}}) => {
  await readable
      .pipeThrough(new TransformStream(transform))
      .pipeThrough(new SFrameTransform({role: "encrypt"}))
      .pipeTo(writable);
};
```
Receiver-side `SFrameTransform({role: "decrypt"}))` would come before `transform`. 

But this only works per-frame, not per-packet. It's incompatible with existing JS transforms, and incompatible with Model A (since the receiver-side JS transform is fed sframe-encrypted frames.

Why does Model B need a switch? I dunno, but this was suggested in our [most recent meeting](https://docs.google.com/presentation/d/19txOdIoxN6SWEyeSyvE8mIkwzOSoejjG06xSOBayzZA/edit?slide=id.g3680465ed33_0_31#slide=id.g3680465ed33_0_31), so I've left it in here.

Model B lets JS read and manipulate the sframe-encrypted frame, at the cost of per-packet. Is there a use case for this? Otherwise I'd go with A.

Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/266 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 19 June 2025 00:09:06 UTC