W3C home > Mailing lists > Public > public-webrtc@w3.org > August 2021

[mediacapture-transform] MSTP's VideoFrame lifetime management model assumes a single consumer (#56)

From: Jan-Ivar Bruaroey via GitHub <sysbot+gh@w3.org>
Date: Mon, 02 Aug 2021 20:00:01 +0000
To: public-webrtc@w3.org
Message-ID: <issues.opened-958462292-1627934399-sysbot+gh@w3.org>
jan-ivar has just created a new issue for https://github.com/w3c/mediacapture-transform:

== MSTP's VideoFrame lifetime management model assumes a single consumer ==
`MediaStreamTrackProcessor` [appears to expect](https://w3c.github.io/mediacapture-transform/#video-processing) _consumers_ to call `VideoFrame.close()` manually. Unfortunately, this:
 1. Assumes a single consumer per VideoFrame
 2. Creates a footgun where consumers may forget to call this method, with potentially devastating OS-level side-effects

Ditto `MediaStreamTrackGenerator` which [calls](https://w3c.github.io/mediacapture-transform/#generator) `VideoFrame.close()` implicitly, preventing reuse with another sink. It too:
 1. Assumes a single consumer per VideoFrame

`VideoFrame`s are immutable objects (with a necessary mutable `close()` method because we don't want to wait for GC, a pattern that works well for a single owner).

A benefit of immutability is supposed to be shareability, so we should be able to tee a stream:

const [readable1, readable2] = new MediaStreamTrackProcessor({track}).readable.tee();

await Promise.all([
  readable1.pipeThrough(new MyTransform(640)).pipeTo(sink1),
  readable2.pipeThrough(new MyTransform(320)).pipeTo(sink2)
...and this would work, except it breaks if `MyTransform` calls `VideoFrame.close()`.

IOW, since sources may have multiple sinks, it seems better for sources to own the `VideoFrame`s it doles out, and manage their lifetime.
source -.---> sink1
        `---> sink2

### Proposal

* Have `MediaStreamTrackGenerator` manage lifetime of `VideoFrame`s it passes to immediate sinks.
  * _(and any transform stream that wants to pass through a `VideoFrame` needs to copy or renew it somehow)_
* Stop having `MediaStreamTrackGenerator` call `VideoFrame.close()` and rely on the producer to do this.

Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/56 using your GitHub account

Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config
Received on Monday, 2 August 2021 20:00:03 UTC

This archive was generated by hypermail 2.4.0 : Monday, 2 August 2021 20:00:04 UTC