Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility

Rich Schwerdtfeger
CTO Accessibility Software Group

Ian Hickson <ian@hixie.ch> wrote on 03/30/2011 04:44:40 PM:

> From: Ian Hickson <ian@hixie.ch>
> To: Richard Schwerdtfeger/Austin/IBM@IBMUS
> Cc: public-html@w3.org
> Date: 03/30/2011 04:46 PM
> Subject: Re: feedback requested: Canvas change for improved hit
> testing that also facilitates accessibility
>
>
> (We were recently requested not to cross-post to multiple HTML working
> group mailing lists but I wasn't sure which mailing list to leave so I
> just left the main one. Feel free to forward this to other groups if I am

> mistaken in this.)
>
> On Wed, 30 Mar 2011, Richard Schwerdtfeger wrote:
> >
> > It is an attempt to simplify hit testing in canvas while at the same
> > time providing a vehicle to tell an assistive technology the bounds of
> > the corresponding UI object being drawn on canvas as represented in
> > fallback content.
>
> FWIW the long-term plan for canvas is to introduce a Path primitive that
> can be used for, amongst other things, trivial hit-testing.
>
> It's not clear to me why an AT needs to know the bounds of non-focused
> elements, or indeed how that would work given a canvas where widgets are
> continuously moving or resizing, or where widgets don't really have a
> concrete dimension until they are being interacted with, but maybe the
> best way to progress with that would be to build a small collection of
> concrete testable examples showing how one is envisaging the canvas API
> being used so that we can make sure we are addressing real use cases (and

> to make sure we agree on what the use cases are).
>
Ian,

First, that is great that the trivial hit testing is going to be addressed.
Here are your real use cases:

Every operating system accessibility API today provides the bounding
rectangle for the object. A screen magnifier may wish to move its zoom to a
particular object in response to a user's request. When the zoom occurs it
requests the bounding rectangle and based on the dimensions of the object
and it type it positions the magnifier zoom point relative to the object.
User's don't always zoom to content in response to focus. Take ZoomText,
ZoomText supports both screen reading and screen magnification and they
build a reference to the accessible objects on the screen. A user can
search and move the magnifier to those objects. This is the same for MAGIC
from Freedom Scientific and SuperNova from Dolphin. Desktop objects can be
moved and resized as well. The information is bounding rectangles are
update when that happens.

When screen readers drive Braille devices they need to know when and where
to do what we might call a line break. A Braille device may have only 80
character cells. A screen reader uses the bounding rectangles to determine
where the breaks occur. A user will be asking for the next "line" of text
content that includes other accessibility properties too.

As I indicated earlier the population of the bounding rectangle (in the
canvas case this would need to be a best fit of the corresponding draw path
delineating the bounds of the rendered object) has been used to populate
the bounding rectangle of drawing objects for years in Windows GUIs. This
same bounding rectangle has also been associated with hit testing. So, in
Windows, and for each object you would use a combination of the following
found in the device context for a drawing object:

- Z order
- bounding rectangle
- clipping region

So, this is not new and the fact that you were considering trivial hit
testing means that you agree that we need hit capability in canvas. Like
the desktop, this is a good way to fill in this hole in the accessibility
API. When you use HTML directly we get this for free as the browsers layout
information fills in the information for each accessible object. For
canvas, the fallback content is separate from what is visually drawn.

What is elegant about the approach we are trying to develop here is:

- The hit testing and mouse events that are normally directed only to
canvas can be directed to the fallback DOM element that receives the
keyboard element
- Transformations for the bounding rectangle are handled by canvas
- The HTML spec defines 1:1 mapping for the fallback content to the UI
object. This allows us to tie the bounding rectangle needed for hit testing
to the object. IE was the first to implement the fallback content but we
could not fill in the bounding rectangles needed. This approach would be
consistent with desktop applications.




> --
> Ian Hickson               U+1047E                )\._.,--....,'``.    fL
> http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'