Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

So in current Web technology, we already have two kinds of pull mechanism,
Canvas2D::drawImage and ImageCapture::grabFrame.
And ImageCapture::grabFrame might fit what you want. And there is not push
mechanism in current Web technology. So I would not address pull mechanism
and focus on push mechanism in this specification.


2015-07-29 11:59 GMT+08:00 Mathieu Hofman <>:

>  Canvas drawImage has indeed the potential to generate duplicate frames
> because it's a synchronous pull API.
>  What I'm suggesting is an *asynchronous pull *API that completes only
> when there is a new frame available that is different from the previous
> frame provided.
>  Something similar to the ImageCapture::grabFrame()
> <>
> API that has the clear requirement of only generating a grab event with a
> frame not previously returned (and ensures the source won't be temporarily
> switched into a special "grab photo" mode).
>  Blindly constraining the frameRate is not back-pressure and is not an
> acceptable solution as:
> - The consumption needs might not be known ahead of time and can vary over
> time
> - Some sources such as screen sharing have a variable framerate
>  A combination of Canvas::drawImage
> and CanvasCaptureMediaStream::requestFrame would simply generate a new
> stream that provides no back-pressure to the source. It has no advantages
> over simply skipping frames.
>  Mathieu
>  ------------------------------
> *From:* Chia-Hung Tai []
> *Sent:* Tuesday, July 28, 2015 8:24 PM
> *To:* Mathieu Hofman
> *Cc:*;;
> *Subject:* Re: Add "MediaStream with worker" for video processing into
> the new working items of WebRTC WG
>   For your case, I think combination of [1] and [2] is what you want. One
> of the reason we choose push mechanism is the existing [1] is pull
> mechanism. And there is some limitations in pull mechanism. For example,
> one of problem is processed duplicated frame in video editor case.
>  And is the frameRate properties in [3] +
> |MediaStreamTrack::applyConstraints| not enough to slow down the internal
> production of frames? Thanks.
>  [1]:
> [2]:
> [3]:
>   BR,
> CTai
> 2015-07-29 10:21 GMT+08:00 Mathieu Hofman <>:
>>  I think we're not talking about the same thing. What I'm looking for is
>> a feedback mechanism that can notify the browser that new frames are not
>> needed right now. Simply ignoring produced frames in the consumer app is
>> different.
>> Some sources like screen sharing do not have to produce a constant stream
>> of frames. It's can be quite expensive to capture the screen for no reason.
>>  An asynchronous pull mechanism solves both problems of the app
>> notifying the browser a frame is wanted, thus only producing when the app
>> needs it (backpressure), and providing frames when the browser producer has
>> a new one, avoiding the app having to diff frames or guess framerate.
>>  The browser implementation under the hood doesn't have to slow down the
>> internal production of frames if it cannot or doesn't want to, and can
>> simply drop frames internally, but it can also use the frame request as a
>> signal to match the production of frames to the demand.
>>  Makes sense?
>>  Mathieu
>>  ------------------------------
>> *From:* Chia-Hung Tai []
>> *Sent:* Tuesday, July 28, 2015 6:49 PM
>> *To:* Mathieu Hofman
>> *Cc:*;;
>> *Subject:* Re: Add "MediaStream with worker" for video processing into
>> the new working items of WebRTC WG
>>    Hi, Mathieu,
>>  As for backpressure, you can just use early return strategy to reach
>> what you want. See [1] for an example. In [2], we use a worker pool to
>> reduce the drop frame rate. We don't address too much frame drop mechanism
>> in the specification. The logic behind it is simple. There is no way to
>> design a drop frame mechanism which fulfilled all kind of web developer
>> needs. So the implementation in User Agent could be simple. And Web
>> developer can manage the flow control on their own like [1] and [2]. So I
>> think the case you mentioned could be resolved by skip the event until you
>> get the acked by the remote side.  Thanks!
>> [1]:
>> [2]:
>>   BR,
>> CTai

Received on Wednesday, 29 July 2015 06:47:25 UTC