Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Hi, Mahieu,
I would like to support all use cases if I could. But the problem is I
can't image how to design an async pull APIs and an elegant sample codes to
guarantee process every frame? That will be great if you can show me some
concrete sample codes. I already tried to figure it out for a while.
Guarantee every frame is the most important reason why we choose push
mechanism. The key use case is video editing.
I think at least below use cases we need to take care.
1. Real time camera processing for webrtc or camera recording => video
processor case
2. Real time video analysis case. In this case, we only analysis frame and
don't modify the stream => video monitor case.
3. Video editing case. We need to guarantee frame by frame processing. =>
video processor case
4. Screen sharing case. I think that is what you want. But I am not sure
what exactly what it would look like.

I am not sure how to provide a solution for all those cases by async pull.
Would happy to learn from you.


2015-07-29 15:30 GMT+08:00 Mathieu Hofman <>:

>  I still do not understand what problem is solved by a push mechanism
> that isn't solved by an async pull mechanism. The latter seem to be a
> strict superset of the former.
> Why can't we come up with one solution that solves all the use cases and
> is usable in the context of workers.
> If you want to keep the push semantics, at least consider the addition of
> a pause/resume feature. But IMHO an async request / pull semantic is
> cleaner.
> Mathieu
> On Jul 28, 2015 11:47 PM, Chia-Hung Tai <> wrote:
>  So in current Web technology, we already have two kinds of pull
> mechanism, Canvas2D::drawImage and ImageCapture::grabFrame.
> And ImageCapture::grabFrame might fit what you want. And there is not push
> mechanism in current Web technology. So I would not address pull mechanism
> and focus on push mechanism in this specification.
>   BR,
> CTai
> 2015-07-29 11:59 GMT+08:00 Mathieu Hofman <>:
>>  Canvas drawImage has indeed the potential to generate duplicate frames
>> because it's a synchronous pull API.
>>  What I'm suggesting is an *asynchronous pull *API that completes only
>> when there is a new frame available that is different from the previous
>> frame provided.
>>  Something similar to the ImageCapture::grabFrame()
>> <>
>> API that has the clear requirement of only generating a grab event with a
>> frame not previously returned (and ensures the source won't be temporarily
>> switched into a special "grab photo" mode).
>>  Blindly constraining the frameRate is not back-pressure and is not an
>> acceptable solution as:
>> - The consumption needs might not be known ahead of time and can vary
>> over time
>> - Some sources such as screen sharing have a variable framerate
>>  A combination of Canvas::drawImage
>> and CanvasCaptureMediaStream::requestFrame would simply generate a new
>> stream that provides no back-pressure to the source. It has no advantages
>> over simply skipping frames.
>>  Mathieu
>>  ------------------------------
>> *From:* Chia-Hung Tai []
>> *Sent:* Tuesday, July 28, 2015 8:24 PM
>> *To:* Mathieu Hofman
>> *Cc:*;;
>> *Subject:* Re: Add "MediaStream with worker" for video processing into
>> the new working items of WebRTC WG
>>   For your case, I think combination of [1] and [2] is what you want.
>> One of the reason we choose push mechanism is the existing [1] is pull
>> mechanism. And there is some limitations in pull mechanism. For example,
>> one of problem is processed duplicated frame in video editor case.
>>  And is the frameRate properties in [3] +
>> |MediaStreamTrack::applyConstraints| not enough to slow down the internal
>> production of frames? Thanks.
>>  [1]:
>> [2]:
>> [3]:
>>   BR,
>> CTai
>> 2015-07-29 10:21 GMT+08:00 Mathieu Hofman <>:
>>>  I think we're not talking about the same thing. What I'm looking for
>>> is a feedback mechanism that can notify the browser that new frames are not
>>> needed right now. Simply ignoring produced frames in the consumer app is
>>> different.
>>> Some sources like screen sharing do not have to produce a constant
>>> stream of frames. It's can be quite expensive to capture the screen for no
>>> reason.
>>>  An asynchronous pull mechanism solves both problems of the app
>>> notifying the browser a frame is wanted, thus only producing when the app
>>> needs it (backpressure), and providing frames when the browser producer has
>>> a new one, avoiding the app having to diff frames or guess framerate.
>>>  The browser implementation under the hood doesn't have to slow down
>>> the internal production of frames if it cannot or doesn't want to, and can
>>> simply drop frames internally, but it can also use the frame request as a
>>> signal to match the production of frames to the demand.
>>>  Makes sense?
>>>  Mathieu
>>>  ------------------------------
>>> *From:* Chia-Hung Tai []
>>> *Sent:* Tuesday, July 28, 2015 6:49 PM
>>> *To:* Mathieu Hofman
>>> *Cc:*;;
>>> *Subject:* Re: Add "MediaStream with worker" for video processing into
>>> the new working items of WebRTC WG
>>>    Hi, Mathieu,
>>>  As for backpressure, you can just use early return strategy to reach
>>> what you want. See [1] for an example. In [2], we use a worker pool to
>>> reduce the drop frame rate. We don't address too much frame drop mechanism
>>> in the specification. The logic behind it is simple. There is no way to
>>> design a drop frame mechanism which fulfilled all kind of web developer
>>> needs. So the implementation in User Agent could be simple. And Web
>>> developer can manage the flow control on their own like [1] and [2]. So I
>>> think the case you mentioned could be resolved by skip the event until you
>>> get the acked by the remote side.  Thanks!
>>> [1]:
>>> [2]:
>>>   BR,
>>> CTai

Received on Thursday, 30 July 2015 01:45:48 UTC