[mediacapture-transform] MSTP's VideoFrame lifetime management model assumes a single consumer (#56)

jan-ivar has just created a new issue for https://github.com/w3c/mediacapture-transform:

== MSTP's VideoFrame lifetime management model assumes a single consumer ==
`MediaStreamTrackProcessor` [appears to expect](https://w3c.github.io/mediacapture-transform/#video-processing) _consumers_ to call `VideoFrame.close()` manually. Unfortunately, this:
 1. Assumes a single consumer per VideoFrame
 2. Creates a footgun where consumers may forget to call this method, with potentially devastating OS-level side-effects

Ditto `MediaStreamTrackGenerator` which [calls](https://w3c.github.io/mediacapture-transform/#generator) `VideoFrame.close()` implicitly, preventing reuse with another sink. It too:
 1. Assumes a single consumer per VideoFrame

`VideoFrame`s are immutable objects (with a necessary mutable `close()` method because we don't want to wait for GC, a pattern that works well for a single owner).

A benefit of immutability is supposed to be shareability, so we should be able to tee a stream:

```js
const [readable1, readable2] = new MediaStreamTrackProcessor({track}).readable.tee();

await Promise.all([
  readable1.pipeThrough(new MyTransform(640)).pipeTo(sink1),
  readable2.pipeThrough(new MyTransform(320)).pipeTo(sink2)
]);
```
...and this would work, except it breaks if `MyTransform` calls `VideoFrame.close()`.

IOW, since sources may have multiple sinks, it seems better for sources to own the `VideoFrame`s it doles out, and manage their lifetime.
```
source -.---> sink1
        `---> sink2
```

### Proposal

* Have `MediaStreamTrackGenerator` manage lifetime of `VideoFrame`s it passes to immediate sinks.
  * _(and any transform stream that wants to pass through a `VideoFrame` needs to copy or renew it somehow)_
* Stop having `MediaStreamTrackGenerator` call `VideoFrame.close()` and rely on the producer to do this.


Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/56 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 2 August 2021 20:00:03 UTC