Re: [mediacapture-surface-control] Is gesture forwarding tied to capture controller or to MediaStreamTrack or to DOM objects? (#45)

> None of the use cases require wheel forwarding from arbitrary elements, which I would oppose.

Your own proposal of `{overlay: element}` **amounts** to forwarding from an arbitrary element. It aims to limit forwarding to elements overlaid on top of an actively playing video element, but it fails to do so, because the workaround is trivial:

<p align="center">
  <img width="380" alt="Image" src="https://github.com/user-attachments/assets/74569ca8-fd63-41ff-a47c-b795f13aeeec" />
</p>

```js
dummyVideoUserNeverSees = stream;
realVideoUserSees = stream;
dummyVideoUserNeverSees.forwardGestures(true, {overlay: overlayElement});
```

You have not increased the UA's ability to enforce forwarding-from-overlay-of-active-video. Limitations on Web developers are only useful if they benefit the user, as per the priority of constituencies. `CaptureController.forwardGestures(element)` will be equally safe for the user and less limiting to the Web developer.

> 1. It would translate coordinates correctly off the video element like it should, not the overlay which may be of different size
> 2. It would limit the interactive hit zone to the video element when the overlay is larger
> 3. It would extend the interactive hit zone to the video element if the overlay only partly covers it or moves

That doesn't work if the preview-tile is a HTMLCanvasElement, since the video frames might be painted over just a portion of the canvas. In that case, the user agent rushing in to offer the Web app assistance would be counterproductive.

In fact, it doesn't work for HTMLVideoElement either. If Media Capture Transform is used to resize or process the video and annotate it, there is a mismatch between where the preview is shown and what the user agent scales to. The UA rushing in would again prove counterproductive.

> 4. It would continue to work if the overlay goes away

First, this "benefit" is not unique to your proposed API shape. (To name just two alternatives - forward from a persistent container, or simply call forwardWheel a second time.)

Second, it's arguable that this isn't even a benefit but a footgun. Explicitly calling forwardWheel a second time is much better.

> 5. It would always find the correct capture based off of the current video.srcObject (and ideally only forward while playing)

First, the matching to `srcObject` is equally viable using either API shape.

Second, that matching is NOT viable either way. As I have pointed out multiple times in the past, the connection between the MediaStream or MediaStreamTrack on the one hand, and the captured surface on the other hand, may be severed if Media Capture Transform or other processing APIs are used (as in your [own proposal](https://github.com/w3c/mediacapture-surface-control/issues/45#issuecomment-2588027586) of "effects").

Nobody gains anything from the implicit indication of the captured surface via `srcObject`, instead of its explicit indication through `CaptureController.forwardGestures()`.

> 6. It would support multiple videos

Not unique to this API shape. This question is completely orthogonal to the questions of (i) point of exposure and (ii) valid sources for gesture-forwarding.

> 7. It would support consecutive captures displayed in the same video element without reinitialization

This purported benefit confers no value, and is arguably a footgun with outright negative value.

> 8. It would be be consistent with how we've extended <video> and <canvas> functionality in the past

I'd argue the opposite. No existing API exposed over either of these elements, uses them as a proxy or a handle for affecting changes to the **source** these elements (purportedly) represent. (I am aware of [captureStream](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/captureStream). It is irrelevant.)

> Interesting. But what if there's a third mostly-transparent canvas for something else that also should scroll the tab? The existing API can't handle that.

Forward gestures from a shared parent/container.

> "Backwarding" turned out too complex.

I am glad we agree on this point.

> Instead, we can secure forwarding: the browser compares the synthetic event forwarded to it in the video element against recent trusted events in its input stack (timestamp, offset), and only scrolls if it finds one. The browser can do this, because it controls everything, and is not limited to what can be done through an event handler #67.

You can employ similar restrictions regardless of API shape.

I see absolutely nothing positive in this proposal, which effectively forces applications that use overlay elements, to replicate the abandoned `sendWheel()` approach. This increased complexity benefits nobody.

-- 
GitHub Notification of comment by eladalon1983
Please view or discuss this issue at https://github.com/w3c/mediacapture-surface-control/issues/45#issuecomment-2624617759 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Thursday, 30 January 2025 14:08:53 UTC