Re: [mediacapture-transform] Is using ReadableStream/WritableStream the right approach? (#4)

The drawbacks of VTR/VTW seem like they will apply to *any* callback API, due to basic properties of the problem statement (conversion between tracks and `VideoFrame`s and back):  

1. Since the track is an input in the Callback approach,  such an API cannot be supported in workers without enabling transfer of MediaStreamTracks.  The streams approach does not have this dependency since it relies on transferrable streams. 
2. The streams approach enables the number of buffered `VideoFrame`s to be controlled via the `MaxBufferSize` argument.  If the Callback approach does not include a similar argument, then it will lose data if the previous Callback does not complete in time. 
3. A Callback approach utilizing events to provide access to `VideoFrame`s will encounter issues similar to those encountered in the WebSockets and RTCDataChannel API.  Even if a `MaxBufferSize` argument is provided to limit the number of queued `VideoFrame`s,, timely delivery of data can be disrupted by other events in the event loop. 
4. The Callback approach has fewer opportunities for optimization than the streams approach.  For example, the streams approach could in future offer native transforms of `VideoFrame`s that may be of interest to developers.  These native transforms could potentially utlize an all-GPU pipeline for optimal performance. 

With respect to major criticisms of the current API (e.g. memory leaks and lack of support for mute/unmute), the Callback approach does not appear to be intrinsically better:  

5. A Callback approach for converting tracks to `VideoFrame`s does not intrinsically address the need to free `VideoFrame`s in a timely way so as to avoid stalling capture.  It is thus equivalent to the streams approach in this regard. 
6. A Callback approach also does not intrinsically address how to handle mute/unmute.  One way to handle this is not to fire the Callback (or in the streams approach, to wait to resolve the promise until the track if unmuted).  So the Callback approach seems equivalent to the streams approach in this regard. 


-- 
GitHub Notification of comment by aboba
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/4#issuecomment-881180047 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 16 July 2021 05:06:26 UTC