[whatwg/streams] Allow web devs to synchronize branches with tee()? (#1157)

[MediaStreamTrackProcessor](https://w3c.github.io/mediacapture-transform/#track-processor) exposes a _real-time_ `ReadableStream` of `VideoFrame`s, e.g. from a camera. It'll drop frames on consumers who don't keep up, to keep data current, and also because it may have a finite pool of GPU-backed frames to vend.

This seems to work well, provided consumers leave `highWaterMark` at `1`.

But with `tee()`, if one or both branches lag [they'll drift from each other](https://jsfiddle.net/jib1/Lb097t1m/6/), causing one branch to stop being real-time. This causes buffer bloat buildup, eating up a ton of `VideoFrame`s.

<img src="https://user-images.githubusercontent.com/3136226/128279169-837abdbc-9ef4-44b4-871c-46b54d1be594.png" width=400>

JS may work around it [by counting frames and trying to synchronize the branches](https://jsfiddle.net/jib1/69sv53j2/14/) by making them take the same amount of time (by waiting for the slower branch), but this can be tricky to get right (spot the bug).

<img src="https://user-images.githubusercontent.com/3136226/128279266-24808f68-4d29-4956-84d9-9df9322ae65b.png" width=400>

A `tee({synchronized: true})` option that did this for you might be better.

This would help ensure both branches stay real-time, at the cost of both dropping frames if one of them lags.

This cost might seem undesirable, so let's look at two potential use cases for `tee()`:
 1. Encode a high-res and a low-res version of the same VideoFrame in parallel (using e.g. a WebCodecs transform polyfill)
 2. Show the same processed video in self-view (at e.g. 60 fps), as is encoded + sent over WebTransport (at e.g. 15 fps)

`{synchronized: true}` should solve the first use case out of the box, since the ultimate sink expects both encodings.

The second use case proves trickier, since we don't want to drop frames in the self-view. To do this without buffer-buildup would probably necessitate a frame-dropper transform on the branch destined for WebTransport, to get it down to 15 fps. This would be a transform that consumes higher fps, resolving promises prematurely to do so. There may also be better ways to solve the second use case without `tee()`, like going through a second MediaStreamTrack.

But we can't prevent someone from using `tee()` to try to solve the second use case, so we need to consider it. But importantly, a frame-dropper transform seems like it should be able to coexist well with a `tee({synchronized: true})` option.

Thoughts?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/issues/1157