[mediacapture-transform] Alternative stream-based API (#59)

jan-ivar has just created a new issue for https://github.com/w3c/mediacapture-transform:

== Alternative stream-based API ==
As requested, this is to discuss our upcoming proposal, which I've written up as a [standards document](https://jan-ivar.github.io/mediacapture-transform/#track-readable), with [3 examples](https://jan-ivar.github.io/mediacapture-transform/#examples).

This brief explainer isn't a substitute for that doc or the [slides](https://docs.google.com/presentation/d/1rdaRvl3h3viqvndyJVzwIAPtOpMjx_2u7q7MBHUwbyY/edit#slide=id.gf05ec18f00_2_76), but walks through a [41-line fiddle](https://jsfiddle.net/jib1/6w1ckby9/):

Since tracks are transferable, instead of of creating all tracks ahead of time and transferring their streams, we simply transfer the camera track to the worker:
```js
const stream = await navigator.mediaDevices.getUserMedia({video: {width: 1280, height: 720}});
video1.srcObject = stream.clone();
const [track] = stream.getVideoTracks();
const worker = new Worker(`worker.js`);
worker.postMessage({track}, [track]);
```
...and receive a processed track in return:
```js
  const {data} = await new Promise(r => worker.onmessage);
  video2.srcObject = new MediaStream([data.track]);
};
```
The worker pipes the camera `track.readable` through a video processing step into a writable VideoTrackSource `source`, whose resulting `source.track` it transfers back with postMessage.

```js
// worker.js
onmessage = async ({data: {track}}) => {
  const source = new VideoTrackSource();
  parent.postMessage({track: source.track}, [source.track]);

  await track.readable.pipeThrough(new TransformStream({transform: crop})).pipeTo(source.writable);
};
```
This avoids exposing data on main thread by default. The `source` (and the real-time media pipeline) stays in the worker, while its `source.track` (a control surface) can be transferred to the main thread. `track.clone()` inherits the same source.

This aligns with mediacapture-main's [sources and sinks model](https://w3c.github.io/mediacapture-main/getusermedia.html#the-model-sources-sinks-constraints-and-settings), which separates a source from its track, and makes it easy to extend the source interface later.

The slides go on to show [how clone() and applyConstraints() used in the worker](https://docs.google.com/presentation/d/1rdaRvl3h3viqvndyJVzwIAPtOpMjx_2u7q7MBHUwbyY/edit#slide=id.gf05ec18f00_2_152) help avoid needing the `tee()` function.

Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/59 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 21 September 2021 20:47:58 UTC