RE: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

There are some use case that require being able to send MediaStream(Track) to other contexts, such as multi window interfaces. Chia-Hung's proposals doesn't solve that use case.
I understand the implementation might not be trivial in User Agents, but is there actually a fundamental reason this cannot be done? At the end of the day, MediaStream(Track) objects are simply light wrappers that are linked to an actual source. It shouldn't matter whether that source is captured / received by the user agent in a process attached to the current context, or another process. From what I gather, there has to be some existing cross process sharing done already, at least when 2 independent contexts in separate processes both need access to a device source, especially webcams which are often exclusive use devices.

Regarding backpressure, the only case this would work with Chia-Hung's proposal is if the processing of the frame is done synchronously in the JavaScript event handler. While such synchronous handling has less side effects in a worker than in the main context, it's still not very Web friendly, and might not be possible in some applications. Imagine the case where you want to only start processing the next frame once you know the current frame has been fully sent over a websocket and acked by the remote side. Probably not a fully realistic scenario, but a possible one none-the-less.

Mathieu

________________________________
From: rocallahan@gmail.com [rocallahan@gmail.com] on behalf of Robert O'Callahan [robert@ocallahan.org]
Sent: Tuesday, July 28, 2015 4:05 AM
To: Mathieu Hofman
Cc: public-media-capture@w3.org; public-webrtc@w3.org; Chia-Hung Tai
Subject: Re: Add "MediaStream with worker" for video processing into the new working items of WebRTC WG

[Clearing most of the CC list]

I feel very strongly that we should not support transferring MediaStreams or MediaStreamTracks, especially across Workers. That would add great complexity to the Gecko MediaStream and WebAudio implementations. Martin already mentioned the multiprocess issue. Another issue is that proper memory management for WebAudio nodes and MediaStream(Tracks), e.g. to garbage-collect nodes, media elements or streams that are no longer relevant, is already quite difficult, and extending that across thread boundaries would be a nightmare. I much prefer alternatives like Chia-Hung's current proposal, which don't require that.

Note: as Martin also mentioned, supporting WebAudio or even MediaStreams in a worker is not itself much of a problem, if we want to do that. The key is that all connected nodes and streams should be associated with the same Worker or window.

As for backpressure, as I understand it, Chia-Hung's proposal does support a form of backpressure. If a video processing or monitoring callback is still processing frame N when frame N+1 becomes available, frame N+1 is queued until the callback completes, but the queue only has length 1, so if frame N+2 arrives while N is still being processed, frame N+1 is effectively dropped.

Rob
--
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr  esn