RE: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

As I noted earlier, I'm ok with a stream abstraction as long as async / non-blocking back-pressure is supported.
Since Streams [1] are getting specified for the web, we could consider using them here.

[1] https://streams.spec.whatwg.org/#backpressure

Mathieu

________________________________
From: Michael Froehlich
Sent: Wednesday, July 29, 2015 5:41 AM
To: Mathieu Hofman; Chia-Hung Tai
Cc: robert@ocallahan.org; public-webrtc@w3.org; public-media-capture@w3.org
Subject: RE: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

I personally think that a push mechanism similar to reactive streams [1] could work well. It maintains a push model for the data, but allows for non-blocking back pressure if the recipient isn’t capable of handling them at the speed of the source. It is async and solves some of the issues already mentioned here.

-- Michael

[1] http://www.reactive-streams.org/


From: Mathieu Hofman [mailto:Mathieu.Hofman@citrix.com]
Sent: Wednesday, July 29, 2015 9:30 AM
To: Chia-Hung Tai
Cc: robert@ocallahan.org; public-webrtc@w3.org; public-media-capture@w3.org
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG


I still do not understand what problem is solved by a push mechanism that isn't solved by an async pull mechanism. The latter seem to be a strict superset of the former.

Why can't we come up with one solution that solves all the use cases and is usable in the context of workers.

If you want to keep the push semantics, at least consider the addition of a pause/resume feature. But IMHO an async request / pull semantic is cleaner.

Mathieu
On Jul 28, 2015 11:47 PM, Chia-Hung Tai <ctai@mozilla.com<mailto:ctai@mozilla.com>> wrote:
So in current Web technology, we already have two kinds of pull mechanism, Canvas2D::drawImage and ImageCapture::grabFrame. And ImageCapture::grabFrame might fit what you want. And there is not push mechanism in current Web technology. So I would not address pull mechanism and focus on push mechanism in this specification.

BR,
CTai

2015-07-29 11:59 GMT+08:00 Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>>:
Canvas drawImage has indeed the potential to generate duplicate frames because it's a synchronous pull API.

What I'm suggesting is an asynchronous pull API that completes only when there is a new frame available that is different from the previous frame provided.

Something similar to the ImageCapture::grabFrame()<http://secure-web.cisco.com/1ME5wsiEuLMsaZGzKOwkHLl9g0cKiZ-FuRB2NVcUGndJanf4-KDDZrIv3MD8_vfdkTLZNTMtIyPRv_yRggLfpVB-XpRmNRMAh-YyXS7qFbHmtgfECaBBmoiu_xjFu9qKxFatVUnWoS95TRPBbgM0yobTA1KaAVKX_atdvGi8cfOs4tdgwW-cGvQlfqz6UOo-m/http%3A%2F%2Fwww.w3.org%2FTR%2Fimage-capture%2F#widl-ImageCapture-grabFrame-Promise> API that has the clear requirement of only generating a grab event with a frame not previously returned (and ensures the source won't be temporarily switched into a special "grab photo" mode).

Blindly constraining the frameRate is not back-pressure and is not an acceptable solution as:
- The consumption needs might not be known ahead of time and can vary over time
- Some sources such as screen sharing have a variable framerate

A combination of Canvas::drawImage and CanvasCaptureMediaStream::requestFrame would simply generate a new stream that provides no back-pressure to the source. It has no advantages over simply skipping frames.

Mathieu

________________________________
From: Chia-Hung Tai [ctai@mozilla.com<mailto:ctai@mozilla.com>]
Sent: Tuesday, July 28, 2015 8:24 PM
To: Mathieu Hofman
Cc: robert@ocallahan.org<mailto:robert@ocallahan.org>; public-media-capture@w3.org<mailto:public-media-capture@w3.org>; public-webrtc@w3.org<mailto:public-webrtc@w3.org>
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG
For your case, I think combination of [1] and [2] is what you want. One of the reason we choose push mechanism is the existing [1] is pull mechanism. And there is some limitations in pull mechanism. For example, one of problem is processed duplicated frame in video editor case.

And is the frameRate properties in [3] + |MediaStreamTrack::applyConstraints| not enough to slow down the internal production of frames? Thanks.

[1]: http://secure-web.cisco.com/1Xy4R4-EgriH8N8X1o0Evssv7S9Bl9_cVa6coy_nNzQjWkK4VmEl5ECR1JtgjtzAGW6ZHeBWWJstJjsm4e2bgYwZLa4Jbt94wIikBWC5v5gh-FnWSD7Wl1TseIByxInnx-c0CKEz98QfZKKo6yzFMwFG_cKds35pDL6KxZi_AkIUeWpvdETYo9HckHNWTKHGv/http%3A%2F%2Fwww.w3.org%2FTR%2F2dcontext%2F#drawing-images-to-the-canvas
[2]: http://secure-web.cisco.com/1nvPuu7yc9SySYfgzYOnXHjjZ-Pj3X79pRMoV9u34kDRwzM8D_6G9g8li-SObekTDBwtRWmiGIDYWO2-_ea5GlBTCtuJyr8s93xP5IOq8VhQnNueJB-Tsx7EL4ywbDzlAd4gw2WwC2BiMK0DKy7kJahGjjYWrJssNKUzuvqJQSaeRO1LbRS0VjAfZGHgPr9AI/http%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-fromelement%2F#the-canvascapturemediastream
[3]: http://secure-web.cisco.com/18oBT_DP5wv2Yfi3wSlSm4eedi5ynC56-GD6i4JdZ1bErSJbX28Dd_yFvibLsYCzBOA1-vYCYv4e5m17FqyA7esI9UJaTT8IZUVAgmx-eXc8BghDLOPR7dRExAMGnllMHvKKh4QDo89PT33xCOEj-bS-t94PS1xuVPBuTp6Cc_Q7lIF-GjZbjD1Wfsp-yXBx0/http%3A%2F%2Fwww.w3.org%2FTR%2Fmediacapture-streams%2F#track-constrainable-property-registrations



BR,
CTai

2015-07-29 10:21 GMT+08:00 Mathieu Hofman <Mathieu.Hofman@citrix.com<mailto:Mathieu.Hofman@citrix.com>>:
I think we're not talking about the same thing. What I'm looking for is a feedback mechanism that can notify the browser that new frames are not needed right now. Simply ignoring produced frames in the consumer app is different.
Some sources like screen sharing do not have to produce a constant stream of frames. It's can be quite expensive to capture the screen for no reason.

An asynchronous pull mechanism solves both problems of the app notifying the browser a frame is wanted, thus only producing when the app needs it (backpressure), and providing frames when the browser producer has a new one, avoiding the app having to diff frames or guess framerate.

The browser implementation under the hood doesn't have to slow down the internal production of frames if it cannot or doesn't want to, and can simply drop frames internally, but it can also use the frame request as a signal to match the production of frames to the demand.

Makes sense?

Mathieu
________________________________
From: Chia-Hung Tai [ctai@mozilla.com<mailto:ctai@mozilla.com>]
Sent: Tuesday, July 28, 2015 6:49 PM
To: Mathieu Hofman
Cc: robert@ocallahan.org<mailto:robert@ocallahan.org>; public-media-capture@w3.org<mailto:public-media-capture@w3.org>; public-webrtc@w3.org<mailto:public-webrtc@w3.org>
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG
Hi, Mathieu,
As for backpressure, you can just use early return strategy to reach what you want. See [1] for an example. In [2], we use a worker pool to reduce the drop frame rate. We don't address too much frame drop mechanism in the specification. The logic behind it is simple. There is no way to design a drop frame mechanism which fulfilled all kind of web developer needs. So the implementation in User Agent could be simple. And Web developer can manage the flow control on their own like [1] and [2]. So I think the case you mentioned could be resolved by skip the event until you get the acked by the remote side.  Thanks!

[1]: https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor/control_worker.js
[2]: https://github.com/kakukogou/foxeye-demo/blob/master/Sample_Monitor_Multiple_ProcessorWorker/control_worker.js

BR,
CTai