Re: [mediacapture-surface-control] Is gesture forwarding tied to capture controller or to MediaStreamTrack or to DOM objects? (#45)

I have now had time to speak to Web developers and get their feedback, and they are extremely opposed to limiting to video elements. (My apologies if I gave the wrong impression during the editors’ meeting, btw. The meeting with Web developers occurred thereafter.)

The Web developers I am in contact with implemented annotations using a library.

Without libraries, “backwarding” consists merely of invoking `myDrawOnClick()` from `video.onclick` instead of from `canvas.onclick` – a possible change.

But with libraries, the application can’t rewire the handlers-to-functions mapping, because that mapping is unknown, abstracted away by the library. One could potentially redispatch clones of events, but use of `pointer-events: ‘none’` means the handlers won’t be invoked.

Not to mention how non-ergonomic and error-prone it is to redispatch event-clones, and how it loses the `isTrusted` bit.

---

Let’s re-examine, again, what this limitation to video elements is even for. It was motivated by the expectation that the user agent could then ensure a **faithful representation** of the controlled surface to the user, suppressing gesture-forwarding when the representation is faithless. But this is **demonstrably impossible** with annotations, because by design they imply overlaying the video with arbitrary pixels. No matter how annotations are implemented, the user sees a faithless representation of the captured surface, by definition.

---

> A developer could have multiple \<video\> elements representing the same capture but with different video effects applied

First, I am not aware of any Web applications that employ this theoretical pattern of (i) multiple preview tiles with (ii) effects on top. This hypothetical example does not appear realistic to me.

Second, the hypothetical use-case and the security rationale are in contradiction. Applying effects on top of videos renders the representation inherently faithless.

---

To summarize:
- Limiting to video elements…
   - …does not gain any privacy/security benefits. (As argued on multiple threads.)
   - …does not facilitate any UA-level heuristics that could increase privacy/security. (As argued by this comment - annotations render all video elements faithless.)
   - …breaks some use cases (with libraries), and requires anti-patterns and non-idiomatic code structures (without libraries) to try to bypass the limitation.
- Forwarding from multiple elements is not a requirement; no realistic use case for it has been shown.


-- 
GitHub Notification of comment by eladalon1983
Please view or discuss this issue at https://github.com/w3c/mediacapture-surface-control/issues/45#issuecomment-2599195307 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Friday, 17 January 2025 20:59:42 UTC