Re: [mediacapture-transform] We shouldn't require track transferability (#113)

> > I said tracks are useless on workers except for this API (i.e., mediacapture-transform) which artificially requires them.
> 
> This is not artificial, transferring a track to a worker has real benefits compared to the approach you mention. Let's take the example of a web application wanting to do background blur on a camera feed via a MediaStreamTrackProcessor and a VideoTrackGenerator.
> 

Yes, it is an artificial *requirement*.

If you have a use case where having the track in the worker is useful, then that can be very valid, but it doesn't justify making transferability it a *requirement* for mediacapture-transform.  I didn't say track transferability is an artificial feature. I'm just saying it is an artificial *requirement* for  mediacapture-transform that, in addition, is often detrimental.

> First, lifetime management is easier. When the VideoTrackGenerator track gets stopped, its WritableStream will be closed. The web application can listen to this via its closed promise and call `stop` on the getUserMedia track. Also, stopping the worker will kill both VideoTrackGenerator and getUserMedia track, housekeeping is simpler :) This is less convenient when the WritableStream lives in a different context than the track, web developer will need to post message.
> 

You don't need transferability *as a requirement* to support this use case. 
UAs that support track transferability can perfectly support this use case even if transferability is not a requirement for mediacapture-transform.
BTW, I have never heard of this use case from actual developers, but I am not opposed to it. 

> Second, configuration management. If the getUserMedia track is muted, the web app will likely want to mute the VideoTrackGenerator. Ditto when getUserMedia track is unmuted. If the getUserMedia track is in the same context as VideoTrackGenerator, it is very easy to implement for the web developer. Otherwise, web app has to postMessage.
> 
> This has a real user consequences: a few frames will likely be missed by VideoTrackGenerator when getUserMedia track gets unmuted if the web app has to postMessage. With the worker approach, missing frames would be a bug in the UA implementation.
> 

I don't think this is an actual problem because if the getUserMedia track is muted, it will produce no frames and the VideoTrackGenerator will see no frames.
Still, if you want to support this use case in this manner, there is nothing in this proposal preventing it.
Like I said, you don't need transferability as a requirement to support this. You just need transferability, which no one is opposing.


> The same principle applies to `configurationchange`, `getSettings`, `applyConstraints`. It is much easier for VideoTrackGenerator, MediaStreamTrackGenerator and getUserMedia track to be all in the same context to make use of these APIs.
> 

The same applies. I haven't heard developers request this, but even if it's a useful use case, you don't need transferability as a requirement to support this. You just need transferability.

> Finally, we introduced MediaStreamTrack transferability as a way to cover some longer term use cases (grabbing camera in an iframe but do rendering/processing in another iframe). The current spec is more future-proof from that point of view as well.
> 

That is an actual use case for which I have seen developer demand and IMO is the main value track transferability can provide. This is completely independent of having track transferability as a requirement for mediacapture-transform.

> > User needs come before the needs of web page authors, which come before the needs of user agent implementors, which come before the needs of specification writers, which come before theoretical purity.
> 
> Right, I think user needs will likely be better served with the current API, as described above.

User needs are not better served by having transferability as a requirement.
Eliminating the requirement of transferability does not prevent any of the use cases mentioned before.
For that you just need transferability, not transferability as a requirement for mediacapture-transform.

On the other hand, *requiring* transferability does make things difficult for some common use cases. The most obvious is playing both the gUM and VTG tracks on a video element in a before/after effects view. In this case you no longer have the gUM track on Window and therefore can't play it on an element. The same applies if you want to use any other track sink available only on Window.

>  I tend to agree that track transferability requires more work from UA implementors, but these costs are overweighted by user and web developer benefits.
> 

All the arguments I've seen so far are benefits of transferability as a standalone feature.  None of these benefits are derived from that transferability being a *requirement* for mediacapture-transform.
Moreover, I've presented use cases where transferability as a requirement makes it more difficult to support common use cases.

So, transferability as a standalone feature supports both the use cases you presented and the ones I presented, but transferability as a mediacapture-transform requirement only supports the use cases you presented and fails to properly support the ones I presented. 
It is clear to me that removing the transferability as a requirement for mediacapture-transform better serves the needs of developers.


> > I also doubt an API that ignores developer requirements and concerns by at least one implementor will get us to interop.
> 
> What are the developer requirements that have been ignored? So far, the developer feedback we received is that MSTP and VTG are working fine in Safari.

Here are some developer requirements that are well known to us and which are ignored by the current version of the spec (not all of these are related to the issue we're discussing which is track transferability as a mediacapture-transform requirement):
* Audio support
* Keeping the gUM track on Window while processing on Worker (before/after view and other use cases that require track sinks available only on Window)
* Easy feature detection (without requiring the creation of a Worker)
* Processing on Window.


-- 
GitHub Notification of comment by guidou
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/113#issuecomment-2444590576 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 29 October 2024 15:23:10 UTC