Re: [mediacapture-transform] We shouldn't require track transferability (#113)

> I'm in the process of trying to use Safari's new spec-compliant implementation of MediaStreamTrackProcessor/VideoTrackGenerator right now for a virtual background feature.

Glad to hear you're working on this. You have my sympathy for having to deal with these API differences right now. But this goes away as more browsers implement the standard API. Once all major engines have it, absence of this feature should be the rare exception and not worth optimizing for.

I'd like to help you over the hump here, as that seems more productive than trying to standardize a third API over what amounts to a transient problem.

> - It is indeed annoying to have to feature detect by first creating a Worker. I'm not aware of any other API that is Worker only, ...

[AudioWorkletProcessor](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletProcessor) is an interface only exposed in worklets. But developers never feature-detect it directly. They pragmatically assume it exists if [AudioWorkletNode](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkletNode) exists on main-thread.

Similarly, developers today can test for [MST transferability](https://github.com/w3c/mediacapture-transform/issues/113#issuecomment-2383916093) and assume the worker APIs exist if true. This works reliably in all browsers today, and I think it would be foolish for a future browser to implement MST transfer ahead of implementing the two APIs it's for. Firefox has already committed to not doing so.

Optimizing for a feature NOT existing seems of diminishing value over time.

> - ... We had previously made the assumption that calling applyConstraints on the main thread would always work, but that is no longer the case with this API change. I guess the correct way to solve this would be to postMessage constraints to the worker, ...

Yes, did you try fiddle I posted above to _"[postMessage constraints](https://jsfiddle.net/jib1/hwn4L8kj/)"_ in Safari? The salient part:

```js
function apply(height) {
  worker.postMessage({constraints: {height}});
}
range.oninput = () => apply(number.value = range.value);
number.oninput = () => apply(range.value = number.value);

function work() {
  let track;
  onmessage = async ({data: {before, constraints}}) => {
    if (before) {
      // ...
      track = before;
    } else if (constraints && track) {
      await track.applyConstraints(constraints);
    }
  };
```

> ... this would require a larger change as we would need to have different code paths for browsers that support transferrable tracks and those that don't. ... For new applications this wouldn't be a problem, but it is for us, where we make many assumptions about how tracks are managed.

Thank you for acknowledging that this is not a problem for new applications. I don't think this meets the bar for standardizing a new API. That said, I'd like to help.

[Here](https://jsfiddle.net/jib1/zocw86m3/) is a simple shim that modifies the track's applyConstraints function to do the postMessage. It's a bit ugly perhaps, but this seems required for any drop-in to satisfy a set of (otherwise incompatible) previous assumptions. Hope this helps.

-- 
GitHub Notification of comment by jan-ivar
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/113#issuecomment-2574051986 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 6 January 2025 22:40:47 UTC