Re: [w3ctag/design-reviews] Incubation: [HTML] html-in-canvas (Issue #1204)

progers left a comment (w3ctag/design-reviews#1204)

> We are very supportive of the goals of this proposal, which is well written. It's good to see privacy considerations addressed. But the proposal requires more work.
> 
> This is just initial feedback, which has three parts: (1) specific questions; (2) general concerns around accessibility; and (3) some thoughts on how this may address concerns we have with other proposals.
> 
> # Specifics
> 1. What changes can trigger the paint event? The explainer says when "any canvas children has changed" but then claims "changing the transform does not cause the paint event to fire". This sounds handwavy to me, for example:
>    
>    * What if the transform on a non-direct descendant changes?
>    * What if the scroll offset of a descendant changes?
>    * What about another canvas? This might lead to a ton of other issues though
>    * And there might be an interop risk. For certain types of changes, a UA implementation may keep them completely off the main thread, and doesn't even run update the rendering steps for them. Examples include:
>      
>      * Video
>      * Animated images
>      * A blinking caret (as in the basic example)

Blinking cursors, transform changes on non-direct descendants, video frames, animated images, and nested canvas changes would all cause the paint event to fire. The developer's javascript is ultimately responsible for updating the canvas rendering, so browsers are unable to do optimizations which avoid javascript in the proposal, but we have considered a future addition which can support threaded effects (see: https://github.com/WICG/html-in-canvas#future-considerations-supporting-threaded-effects-with-an-auto-updating-canvas).

> 2. Maintaining the interactability of the child elements seems to be a goal of this proposal, according to:
>    > Browser features like hit testing, intersection observer, and accessibility rely on an element's DOM location. To ensure these work, the element's transform property should be updated so that the DOM location matches the drawn location.
>    
>    
>    This looks very hacky and is probably not enough. For example:
>    
>    * What if the canvas has multiple child elements whose `drawElementImage()` calls are not in the DOM order? This might mess up hit testing even with their `transform` property updated to match the drawn location.
>    * What if the same child element is drawn for multiple times?
>    
>    We'd like to see a more holistic solution.

We are currently iterating on a design to address this. With the changes in https://github.com/WICG/html-in-canvas/issues/134 https://github.com/WICG/html-in-canvas/issues/135, 2d canvas contexts will automatically update the transform and hit test order of drawn elements. The API changes also support drawing elements multiple times without updating hit testing, to enable reflection-like effects. We are currently implementing these changes and hope to have an update for those issues next week.

> 3. Bikeshed idea: the naming of the `paint` event. It sounds too general while its actual meaning is element image changed.
> 
> # Some accessibility concerns
> We are still looking into the various threads relating to the accessibility of the proposal. The overall goal and direction are very welcome.
> 
> We note that there are some significant unresolved issues that could quite significantly alter the accessibility of the end result, the way that the UA and AT need to work, and the DX.
> 
> One example is [WICG/html-in-canvas#134](https://github.com/WICG/html-in-canvas/issues/134) but there are others too. We will come back with further feedback on this and other issues, as appropriate, in due course.

https://github.com/WICG/html-in-canvas/issues/134#issuecomment-5064386187 and https://github.com/WICG/html-in-canvas/issues/134#issuecomment-5210224649 describe what I think is the best way forward given the feedback from AT users and the Aria WG. I hope we will have this implemented in Chromium for testing next week. I'd very much appreciate feedback if there are aspects that the discussion missed, or if you find issues when using the implementation.

> 
> # Some possible wins over concerns raised in other proposals
> There has been a lot of work done on rendering decorative text in the canvas, i.e. https://tag-github-bot.w3.org/gh/w3ctag/design-reviews/1095
> 
> We were [not satisfied with the canvas text metrics proposal](https://github.com/w3ctag/design-reviews/issues/1095#issuecomment-3872160989), because it introduces at first a complex, imperative API that requires the developer to implement accessibility themselves. We would've preferred to see, as is common on the platform, a simpler, declarative API, that has accessibility built in, come along first, so that most developers can just use that.
> 
> We felt that it would be possible to add such a simple declarative API as part of the HTML-in-canvas work. What do you think? This could solve an accessibility barrier whilst supporting more artistic uses of the canvas.
> 
> There have been two recent developments in this area, which we're looking at:
> 
> * A proposal to make text drawing in canvas accessible by exposing the drawn text to AT - there are significant concerns with this (it would be very noisy and potentially unreliable for AT users).
> * [An update to the Text Metrics proposal was shared at a WHATNOT meeting](https://github.com/whatwg/html/issues/10677#issuecomment-4847744218). We're not yet aware of what's changed, but are looking into it.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3ctag/design-reviews/issues/1204#issuecomment-5286930191
You are receiving this because you are subscribed to this thread.

Message ID: <w3ctag/design-reviews/issues/1204/5286930191@github.com>