Re: [mediacapture-surface-control] Is gesture forwarding tied to capture controller or to MediaStreamTrack or to DOM objects? (#45)

> Risks to end-users of wheel-jacking comes ahead of inconveniencing web developers

Restrictions that do not increase the user agent's ability to protect the user are worse than useless. As I have mentioned multiple times, the proposal that wheel gestures should only be forwarded from the video element, does not increase the user's safety - any safeguards that the UA wishes to implement can be equally implemented without this limitation.

> wheel forwarding [...] clearly a feature of the video element presenting it

Or the HTMLCanvasElement presenting the video.
Or the container for either.
Or any other possible way of obtaining the frame, manipulating it (e.g. annotations) and then displaying it to the user.

> 1. coordinates, which need to align with the offset and scaling of the video presentation.
> 2. Resizing or re-layout of the video element will affect the necessary transformation of coordinates to land in the right place in the captured tab.

This is equally correct regardless of which element we forward gestures from; a video element, a canvas element, or a container that indirectly owns either.

> 3. proposed restrictions, like video.srcObject matching a the right capture, and forwarding only working while it is playing (meaning end-of-forwarding should probably be tied to end of playback, or end-of-track at least, not end of capture which is different due to track cloning).

User agents that wish to tie forwarding to playback are welcome to do so; the current API shape does not prevent such UA-specific limitation.

> If the only reason to put it elsewhere is a library not designed to overlap an interactive element, then that is not a good reason.

This is not a specific library; this is a general issue. You have taken an action item to show how it could be solved. I am still waiting for that.

> I'm not opposed to solving the annotation use case, but it's secondary to the primary design of this API in my view.

It is a core requirement, as expressed by Web developers. No indication has ever been given that it is secondary. The API must satisfy this use-case.

> `await videoElement1.forwardGestures(true, {overlay: element}});`

I see two possible ways of understanding this proposal:
1. When `{overlay: element}` is specified, it tells the user agent that relevant gestures should not be consumed by the overlaid `element` and instead be forwarded down to `videoElement1` below. If so, then this appears to be an expansion of `pointer-events: none;` to work for specific event types, and that is out of scope.
2. When `{overlay: element}` is specified, it tells the user agent that gestures should be forwarded from `element` rather than `videoElement1`. In that case, this API behaves identically to `CaptureController.forwardGestures(element)`, which is arguably shorter and clearer.

Please advise which of these two options you meant, or whether there is a third interpretation.

> we can probably discuss whether the {overlay: div} argument is truly needed or if the user agent can figure out the overlapping element(s) itself.

That would be impossible. The video preview tile could have multiple overlays. For example, one mostly-transparent canvas for annotations, and one draggable sticky note; scrolling over the invisible canvas should scroll the captured tab, but scrolling over the sticky note should scroll the sticky note itself.

-- 
GitHub Notification of comment by eladalon1983
Please view or discuss this issue at https://github.com/w3c/mediacapture-surface-control/issues/45#issuecomment-2622211541 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 29 January 2025 17:02:58 UTC