RE: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Hi all,

Moving the discussion to the public-media-capture list as it's probably a better place to discuss this.

>From what I understand Media Capture is a joint effort between the WebRTC and Devices WG, and hasn't had a WG of its own so far. What I find interesting is that more and more features simply relate to the media, independently of whether it's remote or local. Maybe we should seriously consider forming a "Media" WG that can focus on these features, thus leaving the WebRTC WG focus on the remote aspect, and the Device WG focus on the "device sourcing" of the media. Besides this worker proposal, other examples of media spec partly under the WebRTC WG's responsibilities and that would fit the purpose of a dedicated Media WG are the MediaStream Recording or MediaStream Image Capture specs. One could also argue that further work on the Media Source Extensions spec published by the HTML Working Group could be under the responsibility of a new Media group as well, since it seems to be the mirror of the Recording feature.

Regardless of how the work on this done, I'd like to note a couple things regarding this worker proposal:
- MediaStream(Track)s as objects should be transferable to other contexts, them being worker of any kind, or other window contexts. See discussion on the public media capture list circa May 20 entitled "[Exposed] attribute on our APIs"<https://lists.w3.org/Archives/Public/public-media-capture/2015May/thread.html#msg63> and previous discussions referenced in there. Once the MediaStream(Track) object is transferred to the contexts of your choice (or kept in the main thread if that's the strange choice of the app), you could create a "VideoProcessor" object directly from the MediaStreamTrack in that context.
- I like the approach of a "push" model, especially for source with variable framerate such as screen capture. However there should also be a way to put "backpressure" in case the updates cannot be processed or are not required by the application. Some sources (such as screen sharing) might be able to slow down their production of frames in this case, releasing computer resources. Maybe we should consider the usage of Streams<https://streams.spec.whatwg.org/>? I'm actually wondering if the ImageCapture::grabFrame()<http://w3c.github.io/mediacapture-image/index.html> method might not already solve this. The returned promise should resolve only when a new frame is available, thus avoiding the need to guess the framerate, while allowing the app to request new frames when it actually needs them.

Mathieu

________________________________________
From: Martin Thomson [martin.thomson@gmail.com]
Sent: Monday, July 27, 2015 3:05 AM
To: Rob Manson
Cc: Eric Rescorla; Chia-Hung Tai; public-webrtc@w3.org; Stefan Håkansson LK; Harald Tveit Alvestrand; Dominique Hazael-Massieux; Robert O'Callahan; Ehsan Akhgari; David Baron; Eric Rescorla; Randell Jesup; Kuo, Tzu-Hao; Martin Thomson; Kostiainen, Anssi; Ningxin Hu
Subject: Re: Add "MediaStream with worker" for video processing into the new  working items of WebRTC WG

On 27 July 2015 at 11:43, Rob Manson <roBman@buildar.com> wrote:
> I would think setting up a whole new WG for this work may be prohibitive.

This is something that the IETF struggled with for a time, then fixed.
It was a pretty substantial process improvement.

The artifacts that the media capture task force created are now free
for others to work on; the task force doesn't have to be the sole
place we work on this stuff.  It doesn't scale otherwise.

________________________________________
From: Rob Manson [roBman@buildAR.com]
Sent: Monday, July 27, 2015 2:43 AM
To: Eric Rescorla; Chia-Hung Tai
Cc: public-webrtc@w3.org; Stefan Håkansson LK; Harald Tveit Alvestrand; Dominique Hazael-Massieux; Robert O'Callahan; Ehsan Akhgari; David Baron; Eric Rescorla; Randell Jesup; Kuo, Tzu-Hao; Martin Thomson; Kostiainen, Anssi; Ningxin Hu
Subject: Re: Add "MediaStream with worker" for video processing into the new  working items of WebRTC WG

Hi Eric,

I understand the need for the WG to focus on all the work required for
the core WebRTC use case of peer-to-peer communication (e.g. video
conferencing, data sharing, etc).

But part of our new charter does explicitly describe this type of work.

   Video Stream Functions
   An extension of the Media Stream Functions to
   process video streams, to enable features such as
   bandwidth limiting, image manipulation or "video mute".
   http://secure-web.cisco.com/1n43-rPmSYFrGmsyfA4d4Ulv8GgpKweI4yyw352H_dtUz0VncfXrFyJySRh6P7GsZpN5JiHGouavwCS88cpAYZO-7adguUPtwo5jdFC7LMYIcFGDYJwcH5hNdmptlyiKq1rbJnjP_qcgS48w_nQYy92h5hXb6dmFMBVXyVPVJWqh4DmMYTRlUdzkFtEtNxrYm/http%3A%2F%2Fwww.w3.org%2F2015%2F07%2Fwebrtc-charter.html

e.g. ...process video streams, to enable...image manipulation...

I'm happy to be contradicted by the W3C staff - but I would think
setting up a whole new WG for this work may be prohibitive. Dom?

And by this work's very nature it will definitely relate to Timed Media,
HTML (MediaElement), CanvasContext2D and WebGL. But I think getUserMedia
is at the heart of this and it has already stimulated a lot of
productive discussion related to the Media Capture and Streams Depth
Extension too. This is also likely to help flesh out the Camera  Control
API too.

But however it proceeds I'd like to positively support the need for this
work - wherever it fits.

roBman

________________________________
From: Eric Rescorla [ekr@rtfm.com]
Sent: Monday, July 27, 2015 2:25 AM
To: Chia-Hung Tai
Cc: public-webrtc@w3.org; Stefan Håkansson LK; Harald Tveit Alvestrand; Dominique Hazael-Massieux; Robert O'Callahan; Ehsan Akhgari; David Baron; Eric Rescorla; Randell Jesup; Kuo, Tzu-Hao; Martin Thomson; Kostiainen, Anssi; Ningxin Hu; Rob Manson
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Chia-Hung,

I do not support adding this item to the work of the WebRTC WG. The WG
has an enormous amount to do already and this is largely orthogonal to the
major thrust of our efforts, which is to enable real-time applications (e.g.,
video conferencing.)

It's a particularly bad idea to form a task force for this. While perhaps
necessary at the time for the specific case of getUserMedia, it has been
a real hassle to have fragmented discussion of WebRTC-related topics.

If you want to pursue standardization of this technology, you should form
a separate W3C WG. You might also consider taking it to the
Timed Media WG.

-Ekr

________________________________
From: Chia-Hung Tai [ctai@mozilla.com]
Sent: Monday, July 27, 2015 2:12 AM
To: public-webrtc@w3.org; Stefan Håkansson LK; Harald Tveit Alvestrand; Dominique Hazael-Massieux; Robert O'Callahan; Ehsan Akhgari; David Baron; Eric Rescorla; Randell Jesup; Kuo, Tzu-Hao; Martin Thomson; Kostiainen, Anssi; Ningxin Hu; Rob Manson
Subject: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

Hi, Chairs, Dom and all,

This is Chia-hung Tai from Mozilla. I would like to add a new working items, "MediaStream with worker"[1] into WebRTC WG and form a new task force for this specification. This working item is based on a project called FoxEye[2]. You can check the use cases in [3].

This specification extends the Media Capture and Streams specification to allow JavaScript developers to process video frame data in workers on the web applications.

I already implement a prototype based on the spec[1]. And get positive feedbacks from Mozilla DOM peers. I think it should be standardized. You can see the demo in [4]. Looking forward to hear your voices. Thanks

[1]: http://chiahungtai.github.io/mediacapture-worker/
[2]: https://wiki.mozilla.org/Project_FoxEye
[3]: https://wiki.mozilla.org/Project_FoxEye#Use_Cases
[4]: https://www.youtube.com/watch?v=prybkXsTGXY

BR,
CTai