Re: WebRTC in workers

> Le 22 juin 2018 à 2:25 AM, Harald Alvestrand <harald@alvestrand.no> a écrit :
> 
> Den 22. juni 2018 05:00, skrev youenn fablet:
>> Hi all,
>> 
>> Following on last F2F meeting, please find below some thoughts on WebRTC in workers.
>> This should complete my related action item.
>> 
>> Exposing WebRTC constructs to workers (including service workers) would benefit some scenarios. The scenarios envisioned today mostly make use of data channel as workers currently lack APIs to handle media content. Some identified benefits are:
>> - Having a single and more persistent WebRTC connection if done in service worker
>> - Having a more flexible connection than WebSocket
>> - Processing data out of the main thread
>> 
>> WebRTC in workers can be split in different items:
>> 1. Enable data channel support in workers
>> 2. Enable WebRTC media exchange support in workers
>> 3. Enable transferring WebRTC/media objects between different contexts
>> 4. Enable processing media content in workers
>> 
>> Item 1 can be implemented on top of WebRTC 1.0 by exposing the following interfaces to workers, each interface being added a constructor as defined more or less like in ORTC:
>> - RTCIceTransport
>> - RTCDtlsTransport
>> - RTCSctpTransport
>> - RTCDataChannel
>> This might allow shimming PeerConnection for data channel use.
>> 
>> Item 2 would require exposing these additional interfaces:
>> - RTCRtpSender/RTCRtpReceiver + ORTC-like constructors
>> - MediaStreamTrack/MediaStream
>> Combined with item1, it might be envisioned to expose RTCPeerConnection natively or as a shim.
>> 
>> Item 3 covers the possibility to transfer MediaStream/MediaStreamTrack between different contexts (workers, cross origin documents) living in potentially different threads/processes.
>> It could also cover the possibility to transfer RTC objects between different contexts.
>> 
>> Item 4 exposes functionality to process/generate MediaStreams.
>> WebAudio could be used to generate/process audio tracks.
>> OffscreenCanvas could be used as a basis to generate/process video tracks.
>> 
>> Item 1 seems valuable as an incremental improvement over WebRTC 1.0.
>> Items 2, 3 and 4 need more study in terms of use cases, feasibility…
>> 
>> Thanks,
>>  Y
>> 
> 
> My immediate thought is that item 4 can be done with only item 3: If I
> can transfer a MediaStreamTrack into a worker, and have the APIs for the
> MediaStreamTrack exposed there, I can do processing of media content in
> workers (MediaStreamTrack in / MediaStreamTrack out).

True that item 4 makes more sense if item 3 is available.
If item 3 is not available, but item 2 is, we could envision the case of a connection created in a worker that would generate some MediaStreamTrack to be processed in the worker.
Based on the processing, the worker could post messages to update the UI.

Agreed though that transferring a MediaStreamTrack would unlock more use cases and seems natural.
In terms of implementation, closing the context in which would be created the media stream track would end the transferred media stream track.

> 
>> From the description of what you note as requirements for item 1, it
> sounds as if you are imagining that the worker would be able to set up
> its own data channels with all the transport stuff; another option is to
> make RTCDataChannel transferable, and let PeerConnection remain in [window].

This does not cover the case of long lasting connections in shared/service workers: closing the peer connection window would close the data channel.
Note also that transferring a RTCDataChannel between different contexts is already shimable using postMessage.

> So I'd like to split the scenario classification along the axis of
> "window sets up connection" and "worker sets up connection" - and see
> where our scenarios end up in such a matrix.

I can do that. As noted above, the main issue is the lifetime of the various contexts.

Received on Saturday, 23 June 2018 00:20:55 UTC