RE: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

I think we're not talking about the same thing. What I'm looking for is a feedback mechanism that can notify the browser that new frames are not needed right now. Simply ignoring produced frames in the consumer app is different.
Some sources like screen sharing do not have to produce a constant stream of frames. It's can be quite expensive to capture the screen for no reason.

An asynchronous pull mechanism solves both problems of the app notifying the browser a frame is wanted, thus only producing when the app needs it (backpressure), and providing frames when the browser producer has a new one, avoiding the app having to diff frames or guess framerate.

The browser implementation under the hood doesn't have to slow down the internal production of frames if it cannot or doesn't want to, and can simply drop frames internally, but it can also use the frame request as a signal to match the production of frames to the demand.

Makes sense?

Mathieu
________________________________
From: Chia-Hung Tai [ctai@mozilla.com]
Sent: Tuesday, July 28, 2015 6:49 PM
To: Mathieu Hofman
Cc: robert@ocallahan.org; public-media-capture@w3.org; public-webrtc@w3.org
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Hi, Mathieu,
As for backpressure, you can just use early return strategy to reach what you want. See [1] for an example. In [2], we use a worker pool to reduce the drop frame rate. We don't address too much frame drop mechanism in the specification. The logic behind it is simple. There is no way to design a drop frame mechanism which fulfilled all kind of web developer needs. So the implementation in User Agent could be simple. And Web developer can manage the flow control on their own like [1] and [2]. So I think the case you mentioned could be resolved by skip the event until you get the acked by the remote side.  Thanks!

[1]: https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor/control_worker.js
[2]: https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor_Multiple_ProcessorWorker/control_worker.js

BR,
CTai