Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility

Seeing as nobody has commented can we assume that developers have no
problem with our totally changing the canvas 2D API to support clickable
regions?

I would like to change all the canvas 2D API drawing calls to refer to an
optional fallback element. The default would be the canvas element itself.
This would allow canvas to be able to track what was last drawn placing the
last element drawn at the top of the last drawn list per below. If a
clickable region is set for the element, canvas can decide to pass the
click, mouse over, etc. event to the last drawn element whose clickable
region is the pointing position resides within.

Rich

Rich Schwerdtfeger
CTO Accessibility Software Group



From:	Richard Schwerdtfeger/Austin/IBM@IBMUS
To:	public-html@w3.org, public-canvas-api@w3.org,
            public-html-a11y@w3.org
Cc:	franko@microsoft.com, chuck@jumis.com,
            cyns@exchange.microsoft.com, faulkner.steve@gmail.com,
            david.bolter@gmail.com
Date:	03/29/2011 01:20 PM
Subject:	feedback requested: Canvas change for improved hit testing that
            also  facilitates accessibility
Sent by:	public-html-request@w3.org



As part of the canvas accessibility work we need to provide assistive
technologies with bounding rectangles for fallback content that has a 1:1
mapping to visual objects on canvas. This information would be provided to
platform accessibility API to support:

- Screen Magnifiers to zoom to semantic objects in the fallback content
based on their visual rendering in canvas
- Screen readers to make decisions on what gets sent to refreshable Braille
devices (where the line breaks occur)

When standard HTML is used to render content the user agent uses the layout
engine to compute this information but because canvas rendering is
separated from the actual fallback this cannot be achieved unless there is
an association of the drawing space in the canvas UI for the object
represented in fallback content. The canvas accessibility working group's
solution to the problem is combine what we feel is an improvement on hit
testing in canvas by creating a draw path representation of the clickable
region for the the fallback content and associating them with each other.
This solves two problems:

- When you click on the canvas area the mouse click would go to the
fallback content associated with the clickable region. This also allow the
same fallback node that processes the keyboard input for the canvas
rendering to also receive the pointing device input. A common complaint
about canvas is that the author had to manage all the hit testing.
- The drawing path in canvas will manage any transformations to screen
coordinates that you might otherwise need to compute if you were to use say
CSS.
- Allows the user agent to map the the drawing path for the clickable
region to the accessible object

This concept is not new and in fact has been around GUIs for decades. For
example, in Windows, you create a handle to a device context that has
things like: font information, foreground and background color, bounding
rectangles, and clipping regions. This HDC is used in hit testing and the
bounding rectangle was used to supply the accessibility API bounding
rectangle.

To do this properly we need to address overlapping clickable regions where
the last drawn should get the pointing device input. We could provide an
API to do this (something like setLastDrawn(element) but we feel authors
will get quickly frustrated at having to manage that. Ideally, we would
like to see authors associate each drawing call with a corresponding
element in fallback content for which the default could be the canvas
element itself. This is of course a change to the way canvas apps. are
authored.

We would like to hear feedback from the canvas development community.

This early draft of the approach highlights what we are considering (see
between the ZZZ sections). The setLastDrawn(element) is a placeholder. We
would really prefer that the drawing APIs deal with the computation of the
lastdrawn element when clickable regions overlap. We should also note that
an investigation of the DOM event specification left out any mention as to
how the user agent determines if a mouse click occurs within the drawing
space of an element.

(See attached file: Canvas2DClickableRegion.html)

Thank you,
Rich


Rich Schwerdtfeger
CTO Accessibility Software Group[attachment "Canvas2DClickableRegion.html"
deleted by Richard Schwerdtfeger/Austin/IBM]