Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility

Ian,

So I sent you your use cases which more that adequately address Ian's use
case testing. If the last canvas API proposal submission gets accepted we
have one hole left in the accessibility of canvas and that is the bounding
rectangle information for objects.

The cleanest way to address this, and which will provide the added benefit
to authors would be to tie this to a bounding clickable region for drawing
objects rendered on canvas. Ian likes to tie things that added benefit to
the author. This would do that.

The second use case (lucidart.com) clearly articles that there is enough in
canvas that have taken its use far beyond the intended use that Apple had
(I am referring to Eric Carlson's comments at the last HTML Accessibility
Task Force call). Authors are creating multiple canvas objects to aide with
hit testing and positioning of UI components on canvas. I might also add
that the application allows the user to insert text labels (with a cart
that is yet another canvas element overlayed on top of it) to allow the
user to enter text.

We need a better way to do hit testing and using the bounding path of each
drawing object to map to the bounding rectangle for accessible objects in
the canvas subtree and transfer mouse onclick events, etc. to the
associated fallback content that is also processing the keyboard is the
best strategy for addressing accessibility and adding value for the author.

Oliver mentioned the issue of applying full retained graphics to canvas.
Although we could do this it is not essential.

My response to your request seemed to have gone into a black hole and
frankly Apple created this accessibility problem with canvas so I would
like to here Maciej provide some feedback here.

Rich

Rich Schwerdtfeger
CTO Accessibility Software Group



From:	Richard Schwerdtfeger/Austin/IBM@IBMUS
To:	Ian Hickson <ian@hixie.ch>
Cc:	public-html@w3.org
Date:	03/31/2011 03:39 PM
Subject:	Re: feedback requested: Canvas change for improved hit testing
            that also  facilitates accessibility
Sent by:	public-html-request@w3.org





Rich Schwerdtfeger
CTO Accessibility Software Group

Ian Hickson <ian@hixie.ch> wrote on 03/30/2011 06:49:05 PM:

> From: Ian Hickson <ian@hixie.ch>
> To: Richard Schwerdtfeger/Austin/IBM@IBMUS
> Cc: public-html@w3.org
> Date: 03/30/2011 06:49 PM
> Subject: Re: feedback requested: Canvas change for improved hit
> testing that also facilitates accessibility
>
> On Wed, 30 Mar 2011, Richard Schwerdtfeger wrote:
> >
> > Here are your real use cases: [...]
>
> To clarify, what I meant was I think it would be good to get actual HTML
> pages that show the kinds of issues we're trying to solve. Only by having

> real content can we determine how well an API works. We can't design an
> API in a vacuum.
>
>
> > - The hit testing and mouse events that are normally directed only to
> > canvas can be directed to the fallback DOM element that receives the
> > keyboard element
>
> I don't really know what that would mean. This is the kind of thing for
> which actual HTML pages showing what you mean would be fantastic.
>
I attached a file. This example has two check boxes in fallback content
that have a 1:1 mapping to the canvas rendered version. A magnifier cannot
zoom to it in the UI without knowing where it is or moving keyboard focus
to it. As I said they keyboard focus is not enough.
(See attached file: CanvasEditor.html)

So, that is a simple example. Here is a more complex example from Lucidart:


http://www.lucidchart.com/

In this scenario the author has actually created separate canvas element
for drawing objects that are overlayed over the main diagram drawing space.
No doubt the author found it much easier to handle hit testing and mouse
event processing on each individual canvas drawing object. This is terribly
inefficient. Now, we have not made this accessible but to make it
accessible we would need to associate all the separate canvas elements with
the main canvas element for the drawing background. We could in fact use
the bounding rectangle for the entire drawing object (say it is a decision
drawing object in a flow chart) and the magnifier could zoom in on that but
at large magnification levels (say 10X) we might be looking at just a few
of the characters representing the label within the drawing object but we
have of determining where the text for the label is on the screen so the
magnifier can find it and zoom to it.

I hope this provides adequate use cases to explain the problems which are:

- determining the location and bounds of a distinct object in fallback
content but located on the canvas rendering.
- providing adequate hit testing functionality to avoid an author from
having to calve off drawing objects that should be part of canvas just so
they can be manipulated and moved within the canvas drawing space.

Whatever is done for hit testing, the author must be able to:
- define a drawing path that defines the bounds of the actual drawing
element as shown on canvas
- associate that path as the clickable region for an element in fallback
content so that the author can process both mouse (our touch device) and
keyboard input on the same HTML DOM element representing what is drawn on
the canvas just like the rest of HTML.
- Take the closed path and produce a bounding rectangle for the fallback
object in the accessibility API mapping
- Update the bounding rectangle/clickable region based on where the object
last drawn on the screen
- make it possible for the canvas rendering engine to know what was drawn
last placing objects on a higher "z order" to accommodate direction
directing the pointing device input to the appropriate object.

>
> > - The HTML spec defines 1:1 mapping for the fallback content to the UI
> > object. This allows us to tie the bounding rectangle needed for hit
> > testing to the object.
>
> My concern is that most of the time this won't make sense. The use of
> canvas is something that will typically happen when traditional UI
> paradigms don't work, or are not being used. There might not _be_ a
> bounding box, because the "widget", insofar as there is one, might be
> shattered into many pixels distributed across the canvas, with the pixels

> continually drifting around and just coming together when the user
somehow
> indicates a desire to interact with a particular aspect of the UI.
>
> When canvas is being used in a way that it can just be mapped straight to

> actual UI widgets it almost certainly is being misused -- if a 1:1
mapping
> is possible, then canvas is probably not necessary and one should just
use
> HTML instead. (This is similar to the argument that trying to address
text
> editing accessibility in canvas is misguided, since text editing should
> never be done using canvas in the first place.)
>
> What we need are small HTML pages that show examples of UIs for which
> canvas is appropriate and for which we need accessibility hooks that are
> not yet available. It doesn't make sense to talk about this in the
> abstract without examples to look at.
>
> --
> Ian Hickson U+1047E )\._.,--....,'``. fL
> http://ln.hixie.ch/ U+263A /, _.. \ _\ ;`._ ,.
> Things that are impossible just take longer.
`._.-(,_..'--(,_..'`-.;.'[attachment "CanvasEditor.html" deleted by Richard
Schwerdtfeger/Austin/IBM]