Re: [whatwg/streams] Allow web devs to synchronize branches with tee()? (#1157)

> It's desirable in the first use case I mentioned in the OP, and shouldn't get in the way of dropping frames downstream.

As I understand it, the first use case is this one:

> > 1. Encode a high-res and a low-res version of the same VideoFrame in parallel (using e.g. a WebCodecs transform polyfill)

If it is something equivalent to salsify, the idea is to get the output of both versions and pick one. In that case, the application will synchronise the results of the two operations, synchronized=false and synchronized=true will provide the same results.
If this is something like simulcast, the frame rate is probably not selected as the min of both, but as a parameter to optimise the allocated bandwidth (low res => lower frame rate typically).

The idea behind tee is that we have different consumers processing samples at their own pace.
Buffering is streams solution when consumers have different pace.
synchronized=false introduces consumer stalling as the alternative solution.
My understanding is we in fact want frame dropping as the solution in that specific problem space.

> Exactly. Dropping frames downstream ("renderer and encoder") is a common strategy. A sink resolves promises at its discretion, so the streams spec isn't in the way there.

In both synchronized=true and synchronized=false, the solution to get to the optimal is for the 'slow' consumer to read at the 'fast' consumer pace, and drop frames as needed. The question is then why we should introduce a new option if the actual solution is the same no matter whether option is set to true or false.

Also this solution introduces coupling between the consumers which is something we do not want.
This also prevents any backpressure signal from the 'slow' consumer, although the processing pipe might sometimes benefit those signals.

> As a baseline, in the sunny case, where both branches are within the time budget of the source frame rate, both branches wait for the next (fresh) frame (i.e. in sync), and this is entirely normal and not "slow".

In that case, synchronized=true or synchronized=false have the same result.
In practice though, the baseline is probably more something like 'render + encode', with different processing pace.

Say we use a renderer and an encoder and are using tee as these are two different consumers.
Renderer processes at 60 fps, encoder processes at 30 fps.
Encoder is instructed to read as fast as possible and drop frames as needed.
Suddenly, renderer stops rendering as the local view is no longer visible say.
Application will tell the encoder to read frames at its own pace and ask renderer (which is not running) to read frames as fast as possible even though it is not using any of those frames. Or application will have to recreate its own pipeline whenever local view is switched on/off.
To remove coupling, streams would need to deal with frame dropping themselves, which I think we are hesitant to do.

Please also look at the downsides of introducing a new option in terms of complexity/learning curve/debuggability.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/issues/1157#issuecomment-895076255