Re: [mediacapture-transform] We shouldn't require track transferability (#113)

> I said tracks are useless on workers except for this API (i.e., mediacapture-transform) which artificially requires them.

This is not artificial, transferring a track to a worker has real benefits compared to the approach you mention.
Let's take the example of a web application wanting to do background blur on a camera feed via a MediaStreamTrackProcessor and a VideoTrackGenerator.

First, lifetime management is easier.
When the VideoTrackGenerator track gets stopped, its WritableStream will be closed. The web application can listen to this via its closed promise and call `stop` on the getUserMedia track.
Also, stopping the worker will kill both VideoTrackGenerator and getUserMedia track, housekeeping is simpler :)
This is less convenient when the WritableStream lives in a different context than the track, web developer will need to post message.

Second, configuration management.
If the getUserMedia track is muted, the web app will likely want to mute the VideoTrackGenerator.
Ditto when getUserMedia track is unmuted.
If the getUserMedia track is in the same context as VideoTrackGenerator, it is very easy to implement for the web developer.
Otherwise, web app has to postMessage.

This has a real user consequences: a few frames will likely be missed by VideoTrackGenerator when getUserMedia track gets unmuted if the web app has to postMessage. With the worker approach, missing frames would be a bug in the UA implementation.

The same principle applies to `configurationchange`, `getSettings`, `applyConstraints`.
It is much easier for VideoTrackGenerator, MediaStreamTrackGenerator and getUserMedia track to be all in the same context to make use of these APIs.

Finally, we introduced MediaStreamTrack transferability as a way to cover some longer term use cases (grabbing camera in an iframe but do rendering/processing in another iframe). The current spec is more future-proof from that point of view as well.

> User needs come before the needs of web page authors, which come before the needs of user agent implementors, which come before the needs of specification writers, which come before theoretical purity.

Right, I think user needs will likely be better served with the current API, as described above.
I tend to agree that track transferability requires more work from UA implementors, but these costs are overweighted by user and web developer benefits.

> I also doubt an API that ignores developer requirements and concerns by at least one implementor will get us to interop.

What are the developer requirements that have been ignored?
So far, the developer feedback we received is that MSTP and VTG are working fine in Safari.

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/113#issuecomment-2444350270 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 29 October 2024 13:56:58 UTC