Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

For your case, I think combination of [1] and [2] is what you want. One of
the reason we choose push mechanism is the existing [1] is pull mechanism.
And there is some limitations in pull mechanism. For example, one of
problem is processed duplicated frame in video editor case.

And is the frameRate properties in [3] +
|MediaStreamTrack::applyConstraints| not enough to slow down the internal
production of frames? Thanks.

[1]: http://www.w3.org/TR/2dcontext/#drawing-images-to-the-canvas
[2]:
http://www.w3.org/TR/mediacapture-fromelement/#the-canvascapturemediastream
[3]:
http://www.w3.org/TR/mediacapture-streams/#track-constrainable-property-registrations



BR,
CTai

2015-07-29 10:21 GMT+08:00 Mathieu Hofman <Mathieu.Hofman@citrix.com>:

>  I think we're not talking about the same thing. What I'm looking for is
> a feedback mechanism that can notify the browser that new frames are not
> needed right now. Simply ignoring produced frames in the consumer app is
> different.
> Some sources like screen sharing do not have to produce a constant stream
> of frames. It's can be quite expensive to capture the screen for no reason.
>
>  An asynchronous pull mechanism solves both problems of the app notifying
> the browser a frame is wanted, thus only producing when the app needs it
> (backpressure), and providing frames when the browser producer has a new
> one, avoiding the app having to diff frames or guess framerate.
>
>  The browser implementation under the hood doesn't have to slow down the
> internal production of frames if it cannot or doesn't want to, and can
> simply drop frames internally, but it can also use the frame request as a
> signal to match the production of frames to the demand.
>
>  Makes sense?
>
>  Mathieu
>  ------------------------------
> *From:* Chia-Hung Tai [ctai@mozilla.com]
> *Sent:* Tuesday, July 28, 2015 6:49 PM
> *To:* Mathieu Hofman
> *Cc:* robert@ocallahan.org; public-media-capture@w3.org;
> public-webrtc@w3.org
> *Subject:* Re: Add "MediaStream with worker" for video processing into
> the new working items of WebRTC WG
>
>   Hi, Mathieu,
>  As for backpressure, you can just use early return strategy to reach what
> you want. See [1] for an example. In [2], we use a worker pool to reduce
> the drop frame rate. We don't address too much frame drop mechanism in the
> specification. The logic behind it is simple. There is no way to design a
> drop frame mechanism which fulfilled all kind of web developer needs. So
> the implementation in User Agent could be simple. And Web developer can
> manage the flow control on their own like [1] and [2]. So I think the case
> you mentioned could be resolved by skip the event until you get the acked
> by the remote side.  Thanks!
>
> [1]:
> https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor/control_worker.js
> [2]:
> https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor_Multiple_ProcessorWorker/control_worker.js
>
>   BR,
> CTai
>
>