Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

So in current Web technology, we already have two kinds of pull mechanism,
Canvas2D::drawImage and ImageCapture::grabFrame.
And ImageCapture::grabFrame might fit what you want. And there is not push
mechanism in current Web technology. So I would not address pull mechanism
and focus on push mechanism in this specification.

BR,
CTai

2015-07-29 11:59 GMT+08:00 Mathieu Hofman <Mathieu.Hofman@citrix.com>:

>  Canvas drawImage has indeed the potential to generate duplicate frames
> because it's a synchronous pull API.
>
>  What I'm suggesting is an *asynchronous pull *API that completes only
> when there is a new frame available that is different from the previous
> frame provided.
>
>  Something similar to the ImageCapture::grabFrame()
> <http://www.w3.org/TR/image-capture/#widl-ImageCapture-grabFrame-Promise>
> API that has the clear requirement of only generating a grab event with a
> frame not previously returned (and ensures the source won't be temporarily
> switched into a special "grab photo" mode).
>
>  Blindly constraining the frameRate is not back-pressure and is not an
> acceptable solution as:
> - The consumption needs might not be known ahead of time and can vary over
> time
> - Some sources such as screen sharing have a variable framerate
>
>  A combination of Canvas::drawImage
> and CanvasCaptureMediaStream::requestFrame would simply generate a new
> stream that provides no back-pressure to the source. It has no advantages
> over simply skipping frames.
>
>  Mathieu
>
>  ------------------------------
> *From:* Chia-Hung Tai [ctai@mozilla.com]
> *Sent:* Tuesday, July 28, 2015 8:24 PM
> *To:* Mathieu Hofman
> *Cc:* robert@ocallahan.org; public-media-capture@w3.org;
> public-webrtc@w3.org
> *Subject:* Re: Add "MediaStream with worker" for video processing into
> the new working items of WebRTC WG
>
>   For your case, I think combination of [1] and [2] is what you want. One
> of the reason we choose push mechanism is the existing [1] is pull
> mechanism. And there is some limitations in pull mechanism. For example,
> one of problem is processed duplicated frame in video editor case.
>
>  And is the frameRate properties in [3] +
> |MediaStreamTrack::applyConstraints| not enough to slow down the internal
> production of frames? Thanks.
>
>  [1]:
> http://secure-web.cisco.com/1Xy4R4-EgriH8N8X1o0Evssv7S9Bl9_cVa6coy_nNzQjWkK4VmEl5ECR1JtgjtzAGW6ZHeBWWJstJjsm4e2bgYwZLa4Jbt94wIikBWC5v5gh-FnWSD7Wl1TseIByxInnx-c0CKEz98QfZKKo6yzFMwFG_cKds35pDL6KxZi_AkIUeWpvdETYo9HckHNWTKHGv/http%3A%2F%2Fwww.w3.org%2FTR%2F2dcontext%2F#drawing-images-to-the-canvas
> [2]:
> http://secure-web.cisco.com/1nvPuu7yc9SySYfgzYOnXHjjZ-Pj3X79pRMoV9u34kDRwzM8D_6G9g8li-SObekTDBwtRWmiGIDYWO2-_ea5GlBTCtuJyr8s93xP5IOq8VhQnNueJB-Tsx7EL4ywbDzlAd4gw2WwC2BiMK0DKy7kJahGjjYWrJssNKUzuvqJQSaeRO1LbRS0VjAfZGHgPr9AI/http%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-fromelement%2F#the-canvascapturemediastream
> [3]:
> http://secure-web.cisco.com/18oBT_DP5wv2Yfi3wSlSm4eedi5ynC56-GD6i4JdZ1bErSJbX28Dd_yFvibLsYCzBOA1-vYCYv4e5m17FqyA7esI9UJaTT8IZUVAgmx-eXc8BghDLOPR7dRExAMGnllMHvKKh4QDo89PT33xCOEj-bS-t94PS1xuVPBuTp6Cc_Q7lIF-GjZbjD1Wfsp-yXBx0/http%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-streams%2F#track-constrainable-property-registrations
>
>
>
>   BR,
> CTai
>
> 2015-07-29 10:21 GMT+08:00 Mathieu Hofman <Mathieu.Hofman@citrix.com>:
>
>>  I think we're not talking about the same thing. What I'm looking for is
>> a feedback mechanism that can notify the browser that new frames are not
>> needed right now. Simply ignoring produced frames in the consumer app is
>> different.
>> Some sources like screen sharing do not have to produce a constant stream
>> of frames. It's can be quite expensive to capture the screen for no reason.
>>
>>  An asynchronous pull mechanism solves both problems of the app
>> notifying the browser a frame is wanted, thus only producing when the app
>> needs it (backpressure), and providing frames when the browser producer has
>> a new one, avoiding the app having to diff frames or guess framerate.
>>
>>  The browser implementation under the hood doesn't have to slow down the
>> internal production of frames if it cannot or doesn't want to, and can
>> simply drop frames internally, but it can also use the frame request as a
>> signal to match the production of frames to the demand.
>>
>>  Makes sense?
>>
>>  Mathieu
>>  ------------------------------
>> *From:* Chia-Hung Tai [ctai@mozilla.com]
>> *Sent:* Tuesday, July 28, 2015 6:49 PM
>> *To:* Mathieu Hofman
>> *Cc:* robert@ocallahan.org; public-media-capture@w3.org;
>> public-webrtc@w3.org
>> *Subject:* Re: Add "MediaStream with worker" for video processing into
>> the new working items of WebRTC WG
>>
>>    Hi, Mathieu,
>>  As for backpressure, you can just use early return strategy to reach
>> what you want. See [1] for an example. In [2], we use a worker pool to
>> reduce the drop frame rate. We don't address too much frame drop mechanism
>> in the specification. The logic behind it is simple. There is no way to
>> design a drop frame mechanism which fulfilled all kind of web developer
>> needs. So the implementation in User Agent could be simple. And Web
>> developer can manage the flow control on their own like [1] and [2]. So I
>> think the case you mentioned could be resolved by skip the event until you
>> get the acked by the remote side.  Thanks!
>>
>> [1]:
>> https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor/control_worker.js
>> [2]:
>> https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor_Multiple_ProcessorWorker/control_worker.js
>>
>>   BR,
>> CTai
>>
>>
>