Re: [mediacapture-surface-control] Is gesture forwarding tied to capture controller or to MediaStreamTrack or to DOM objects? (#45)

> I see two possible ways of understanding this proposal:

It's 2.

> 2. When `{overlay: element}` is specified, it tells the user agent that gestures should be forwarded from element rather than videoElement1. In that case, this API behaves identically to CaptureController.forwardGestures(element), which is arguably shorter and clearer.

That's incorrect. It would do a lot more than that:
1. It would translate coordinates correctly off the video element like it should, not the overlay which may be of different size
2. It would limit the interactive hit zone to the video element when the overlay is larger
3. It would extend the interactive hit zone to the video element if the overlay only partly covers it or moves
4. It would continue to work if the overlay goes away
5. It would always find the correct capture based off of the current `video.srcObject` (and ideally only forward while playing)
6. It would support multiple videos
7. It would support consecutive captures displayed in the same video element without reinitialization
8. It would be be consistent with how we've extended \<video> and \<canvas> functionality in the past

None of the use cases require wheel forwarding from arbitrary elements, which I would oppose. This addresses that.

> > we can probably discuss whether the {overlay: div} argument is truly needed or if the user agent can figure out the overlapping element(s) itself.
> 
> That would be impossible. ... multiple overlays. For example, one mostly-transparent canvas for annotations, and one draggable sticky note; [only] scrolling over the invisible canvas should scroll the captured tab ...

Interesting. But what if there's a _third_ mostly-transparent canvas for something else that also should scroll the tab? The existing API can't handle that.

My answer to resolve this (and my action) is for the overlays that wish to scroll the video element (and only those) simply forward wheel events to the video element using standard practice.

Here's an example of an overlay forwarding events to a video element: https://jsfiddle.net/jib1/2x584917/

<img width="308" alt="Image" src="https://github.com/user-attachments/assets/3df6e87e-e80d-4fa8-a7cb-2e87afe65bc2" />

 "Backwarding" turned out too complex. Instead, we can secure forwarding: the browser compares the synthetic event forwarded to it in the video element against recent trusted events in its input stack (timestamp, offset), and only scrolls if it finds one. The browser can do this, because it controls everything, and is not limited to what can be done through an event handler #67.

I don't think we should extend web platform APIs to pass in an element just to avoid existing libraries doing that.

-- 
GitHub Notification of comment by jan-ivar
Please view or discuss this issue at https://github.com/w3c/mediacapture-surface-control/issues/45#issuecomment-2623210084 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 30 January 2025 00:05:47 UTC