Re: [mediacapture-transform] We shouldn't require track transferability (#113)

I'm in the process of trying to use Safari's new spec-compliant implementation of MediaStreamTrackProcessor/VideoTrackGenerator right now for a virtual background feature. Our existing app ([https://app.zoom.us/wc](https://app.zoom.us/wc)) handles both Chrome's implementation (we transfer the `readable` to the worker) and the case where MediaStreamTrackProcessor isn't available at all. Here are my thoughts as a user of this new API change:

-  It is indeed annoying to have to feature detect by first creating a Worker. I'm not aware of any other API that is Worker only, and our existing code assumes that all feature detection can be done on the main thread. We would either have to create and then destroy a dummy worker just for feature detection, or significantly modify how our startup flow works in order to feature detect when the real worker is first created.

- The WebRTC/getUserMedia API is for the most part main thread only (with the recent exception of transferrable RTCDataChannels). We had previously made the assumption that calling `applyConstraints` on the main thread would always work, but that is no longer the case with this API change. I guess the correct way to solve this would be to postMessage constraints to the worker, but this would require a larger change as we would need to have different code paths for browsers that support transferrable tracks and those that don't.

Overall, the API change forces us to restructure how we manage tracks by changing how track state is synced between the main thread and worker threads. Previously, all track management is done on the main thread - but with this new API it's now all done in the worker threads. For new applications this wouldn't be a problem, but it is for us, where we make many assumptions about how tracks are managed. I understand that there are benefits for this API be available on workers, but not why it needs to be worker *only*, especially when doing so makes adapting existing applications challenging.

-- 
GitHub Notification of comment by mehagar
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/113#issuecomment-2569916788 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 3 January 2025 23:04:20 UTC