[webrtc-encoded-transform] Current steps are inefficient and difficult to build on. Cleanup needed (#229)

jan-ivar has just created a new issue for https://github.com/w3c/webrtc-encoded-transform:

== Current steps are inefficient and difficult to build on. Cleanup needed ==
[RTCRtpScriptTransformer](https://w3c.github.io/webrtc-encoded-transform/#rtcrtpscripttransformer) holds the only JS-exposed readable + writable pair, which is what the JS app calls `pipeTo` on:
```js
// worker.js
onrtctransform = async ({transformer: {readable, writable, options}}) =>
    await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable);
```

Most of the implementation can be described by specifying how the underlying source and the underlying sink MUST work.

This would help trim a lot of prose from the spec.

In contrast, the current prose describes a JS-shim-like reference implementation, which was helpful at one point. But it prescribes objects that aren't necessary in a C++ implementation. This is proving difficult to build on:

1. The [stream creation](https://w3c.github.io/webrtc-encoded-transform/#stream-creation) steps say to create the following objects on main thread:
   - rtcRtpSender.[[readable]]
   - rtcRtpSender.[[writable]]
   - rtcRtpReceiver.[[readable]]
   - rtcRtpReceiver.[[writable]]
   - And then queues a task to [pipeTo](https://streams.spec.whatwg.org/#readable-stream-pipe-to) with[ this](https://webidl.spec.whatwg.org/#this).[[readable]],[ this](https://webidl.spec.whatwg.org/#this).[[writable]]
   
   These internal objects don't need to exist, and `pipeTo` is a JS API, not infra. Main thread is also not where media encoders/decoders or RTP packetizers are.

2. The [new RTCRtpScriptTransform(worker, options, transfer)](https://w3c.github.io/webrtc-encoded-transform/#dom-rtcrtpscripttransform-rtcrtpscripttransform) steps set up 
    - [rtcRtpScriptTransform](https://w3c.github.io/webrtc-encoded-transform/#rtcrtpscripttransform).[[writable]] to [id](https://streams.spec.whatwg.org/#identity-transform-stream).[[writable]].
    - [rtcRtpScriptTransform](https://w3c.github.io/webrtc-encoded-transform/#rtcrtpscripttransform).[[readable]] to [id](https://streams.spec.whatwg.org/#identity-transform-stream).[[readable]].
    - Then transfers the id streams to the worker where they’re assigned to
        - a new[ RTCRtpScriptTransformer](https://w3c.github.io/webrtc-encoded-transform/#rtcrtpscripttransformer).[[readable]] and [[writable]]

    Here, [rtcRtpSender](https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender)/[rtcRtpReceiver](https://w3c.github.io/webrtc-pc/#dom-rtcrtpreceiver) and [rtcRtpScriptTransform](https://w3c.github.io/webrtc-encoded-transform/#rtcrtpscripttransform) have separate internal readable/writables, and the sender/receiver.[transform](https://w3c.github.io/webrtc-encoded-transform/#attribute) setter steps ties them together.

    This relies on [transferable streams](https://streams.spec.whatwg.org/#transferrable-streams) (a special tunnel-like construct) to carry the weight of the cross-threading part of the implementation. That is AN implementation choice, but hardly the only one, nor the most likely in C++. There's no reason for a C++ implementation to work this way, or use streams internally this way, or rely on transferable streams like this, when we probably don’t want to involve the main thread with this data piping at all.

It seems simpler to express implementation requirements for the one readable and one writable this specification defines directly. Both live and die on the dedicated worker and the UA needs to dispatch data to and from the worker to service the app-created pipe.

Please view or discuss this issue at https://github.com/w3c/webrtc-encoded-transform/issues/229 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 21 May 2024 00:15:53 UTC