RE: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Canvas drawImage has indeed the potential to generate duplicate frames because it's a synchronous pull API.

What I'm suggesting is an asynchronous pull API that completes only when there is a new frame available that is different from the previous frame provided.

Something similar to the ImageCapture::grabFrame()<http://www.w3.org/TR/image-capture/#widl-ImageCapture-grabFrame-Promise> API that has the clear requirement of only generating a grab event with a frame not previously returned (and ensures the source won't be temporarily switched into a special "grab photo" mode).

Blindly constraining the frameRate is not back-pressure and is not an acceptable solution as:
- The consumption needs might not be known ahead of time and can vary over time
- Some sources such as screen sharing have a variable framerate

A combination of Canvas::drawImage and CanvasCaptureMediaStream::requestFrame would simply generate a new stream that provides no back-pressure to the source. It has no advantages over simply skipping frames.

Mathieu

________________________________
From: Chia-Hung Tai [ctai@mozilla.com]
Sent: Tuesday, July 28, 2015 8:24 PM
To: Mathieu Hofman
Cc: robert@ocallahan.org; public-media-capture@w3.org; public-webrtc@w3.org
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

For your case, I think combination of [1] and [2] is what you want. One of the reason we choose push mechanism is the existing [1] is pull mechanism. And there is some limitations in pull mechanism. For example, one of problem is processed duplicated frame in video editor case.

And is the frameRate properties in [3] + |MediaStreamTrack::applyConstraints| not enough to slow down the internal production of frames? Thanks.

[1]: http://secure-web.cisco.com/1Xy4R4-EgriH8N8X1o0Evssv7S9Bl9_cVa6coy_nNzQjWkK4VmEl5ECR1JtgjtzAGW6ZHeBWWJstJjsm4e2bgYwZLa4Jbt94wIikBWC5v5gh-FnWSD7Wl1TseIByxInnx-c0CKEz98QfZKKo6yzFMwFG_cKds35pDL6KxZi_AkIUeWpvdETYo9HckHNWTKHGv/http%3A%2F%2Fwww.w3.org%2FTR%2F2dcontext%2F#drawing-images-to-the-canvas
[2]: http://secure-web.cisco.com/1nvPuu7yc9SySYfgzYOnXHjjZ-Pj3X79pRMoV9u34kDRwzM8D_6G9g8li-SObekTDBwtRWmiGIDYWO2-_ea5GlBTCtuJyr8s93xP5IOq8VhQnNueJB-Tsx7EL4ywbDzlAd4gw2WwC2BiMK0DKy7kJahGjjYWrJssNKUzuvqJQSaeRO1LbRS0VjAfZGHgPr9AI/http%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-fromelement%2F#the-canvascapturemediastream
[3]: http://secure-web.cisco.com/18oBT_DP5wv2Yfi3wSlSm4eedi5ynC56-GD6i4JdZ1bErSJbX28Dd_yFvibLsYCzBOA1-vYCYv4e5m17FqyA7esI9UJaTT8IZUVAgmx-eXc8BghDLOPR7dRExAMGnllMHvKKh4QDo89PT33xCOEj-bS-t94PS1xuVPBuTp6Cc_Q7lIF-GjZbjD1Wfsp-yXBx0/http%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-streams%2F#track-constrainable-property-registrations



BR,
CTai

2015-07-29 10:21 GMT+08:00 Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>>:
I think we're not talking about the same thing. What I'm looking for is a feedback mechanism that can notify the browser that new frames are not needed right now. Simply ignoring produced frames in the consumer app is different.
Some sources like screen sharing do not have to produce a constant stream of frames. It's can be quite expensive to capture the screen for no reason.

An asynchronous pull mechanism solves both problems of the app notifying the browser a frame is wanted, thus only producing when the app needs it (backpressure), and providing frames when the browser producer has a new one, avoiding the app having to diff frames or guess framerate.

The browser implementation under the hood doesn't have to slow down the internal production of frames if it cannot or doesn't want to, and can simply drop frames internally, but it can also use the frame request as a signal to match the production of frames to the demand.

Makes sense?

Mathieu
________________________________
From: Chia-Hung Tai [ctai@mozilla.com<mailto:ctai@mozilla.com>]
Sent: Tuesday, July 28, 2015 6:49 PM
To: Mathieu Hofman
Cc: robert@ocallahan.org<mailto:robert@ocallahan.org>; public-media-capture@w3.org<mailto:public-media-capture@w3.org>; public-webrtc@w3.org<mailto:public-webrtc@w3.org>
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Hi, Mathieu,
As for backpressure, you can just use early return strategy to reach what you want. See [1] for an example. In [2], we use a worker pool to reduce the drop frame rate. We don't address too much frame drop mechanism in the specification. The logic behind it is simple. There is no way to design a drop frame mechanism which fulfilled all kind of web developer needs. So the implementation in User Agent could be simple. And Web developer can manage the flow control on their own like [1] and [2]. So I think the case you mentioned could be resolved by skip the event until you get the acked by the remote side.  Thanks!

[1]: https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor/control_worker.js
[2]: https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor_Multiple_ProcessorWorker/control_worker.js

BR,
CTai