Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility

We need to able to bind a path to hit testing functionality. It is not
enough just to be able to define and save a path. Clearly, people are
wanting hit testing support.

Rich Schwerdtfeger
CTO Accessibility Software Group



From:	Charles Pritchard <chuck@jumis.com>
To:	Richard Schwerdtfeger/Austin/IBM@IBMUS
Cc:	Boris Zbarsky <bzbarsky@mit.edu>, "cyns@exchange.microsoft.com"
            <cyns@exchange.microsoft.com>, "david.bolter@gmail.com"
            <david.bolter@gmail.com>, "faulkner.steve@gmail.com"
            <faulkner.steve@gmail.com>, "franko@microsoft.com"
            <franko@microsoft.com>, Oliver Hunt <oliver@apple.com>,
            "public-canvas-api@w3.org" <public-canvas-api@w3.org>,
            "public-canvas-api-request@w3.org"
            <public-canvas-api-request@w3.org>, "public-html@w3.org"
            <public-html@w3.org>, "public-html-a11y@w3.org"
            <public-html-a11y@w3.org>, "public-html-request@w3.org"
            <public-html-request@w3.org>
Date:	04/01/2011 05:00 PM
Subject:	Re: feedback requested: Canvas change for improved hit testing
            that also     facilitates accessibility



With VoiceOver, I have to use a bunch of ugly hacks. That happens in
compatibility work.
Hopefully my attendance to the standards mailing list, and promotion of bug
reports, will
reduce the need to use such hacks in the future.

On the thread-- sharing path data in the context of a11y.

Once we start keeping paths around, we're going to encounter push-back to
the next version of canvas.
The existing canvas spec allows-for the recording of all paths, but no
implementers are doing it.

I'm still struggling for a solution which implementers might approach in
this version.
I'm trying to round up some ideas from other canvas-users.

These are the ones I have:

Using CSS pointer-events would be an appropriate way to -enable- slower
code paths.

pointer-events: stroke would work fine with strokeText and stroke(),
and when enabled, may stash pixels in a 1-bit bitmap, and/or a character
buffer.

When pointer-events are set the implementation would expose the current
path
to the dom or canvas2d api.

These are steps towards a long-term solution.

They both help canvas authors, pointer-events really help with DOM
management and speed-up hit testing [by memory trade-off],
and grabbing a DOMString of the path makes debugging easier.

It's reasonable to expect an AT to calculate the extents of a DOMString,
normalized SVG Path.

Those two items provide a foundation for a a few simple methods
to notify the AT that pointer/coordinate focus should be associated with
fallback content.

I'd like to see if they have any support.

-Charles


On 3/30/2011 1:54 PM, Richard Schwerdtfeger wrote:


      Charles, right now the fallback content in canvas is a black hole in
      webkit and thus Safari. So, VoiceOver can't read anything in canvas
      without a Ouija board. (sorry, a little humor is due)


      Rich Schwerdtfeger
      CTO Accessibility Software Group

      Inactive
          hide details for Charles Pritchard ---03/30/2011
      02:24:58
          PM---Oliver, What's your take on the VoiceOver
      integration wCharles Pritchard ---03/30/2011 02:24:58 PM---Oliver,
      What's your take on the VoiceOver integration with canvas?

      From: Charles Pritchard <chuck@jumis.com>
      To: Oliver Hunt <oliver@apple.com>
      Cc: Richard Schwerdtfeger/Austin/IBM@IBMUS, Boris Zbarsky
      <bzbarsky@mit.edu>, "cyns@exchange.microsoft.com"
      <cyns@exchange.microsoft.com>, "david.bolter@gmail.com"
      <david.bolter@gmail.com>, "faulkner.steve@gmail.com"
      <faulkner.steve@gmail.com>, "franko@microsoft.com"
      <franko@microsoft.com>, "public-canvas-api@w3.org"
      <public-canvas-api@w3.org>, "public-canvas-api-request@w3.org"
      <public-canvas-api-request@w3.org>, "public-html@w3.org"
      <public-html@w3.org>, "public-html-a11y@w3.org"
      <public-html-a11y@w3.org>, "public-html-request@w3.org"
      <public-html-request@w3.org>
      Date: 03/30/2011 02:24 PM
      Subject: Re: feedback requested: Canvas change for improved hit
      testing that also facilitates accessibility





      Oliver,

      What's your take on the VoiceOver integration with canvas?
      First and foremost, we're looking to enable authors to provide more
      information to accessibility software.

      This is a brainstorming thread.

      Ideas welcome.

      On Mar 30, 2011, at 12:20 PM, Oliver Hunt <oliver@apple.com> wrote:

            On Mar 30, 2011, at 12:06 PM, Richard Schwerdtfeger wrote:

                  Oliver,

                  It is an attempt to simplify hit testing in canvas while
                  at the same time providing a vehicle to tell an assistive
                  technology the bounds of the corresponding UI object
                  being drawn on canvas as represented in fallback content.
                  Currently there is no mapping. These bounds are needed by
                  screen magnifiers for zooming and screen readers for
                  Braille support (earlier post).

                  Now the canvas author has to manage all the hit testing.
                  This is a canvas deficiency that should have been in
                  place to start - so grafting seems like an inappropriate
                  response. Consequently, I am also seeing canvas
                  applications that create the equivalent of visio by
                  creating separate canvas elements overlayed on top of
                  another canvas to represent drawing objects. This is very
                  inefficient and will get worse unless something is done.

            This is not a canvas deficiency -- canvas is an immediate mode
            renderer, one of the things you have to handle yourself when
            dealing with _any_ immediate mode renderer is hit detection.
            This is true of Canvas, GDI, CG, raw component painting in
            Java, etc, etc

            If you want hit detection to be done for you, you want a
            retained mode renderer, such as SVG.

            --Oliver



                  Rich Schwerdtfeger
                  CTO Accessibility Software Group

                  <graycol.gif>Oliver Hunt ---03/30/2011 12:11:53 PM---This
                  feels like an attempt to graft retained mode rendering on
                  to canvas, what am i missing? --Olive

                  From: Oliver Hunt <oliver@apple.com>
                  To: Richard Schwerdtfeger/Austin/IBM@IBMUS
                  Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com,
                  cyns@exchange.microsoft.com, david.bolter@gmail.com,
                  faulkner.steve@gmail.com, franko@microsoft.com,
                  public-canvas-api@w3.org, public-html@w3.org,
                  public-html-a11y@w3.org, public-html-request@w3.org
                  Date: 03/30/2011 12:11 PM
                  Subject: Re: feedback requested: Canvas change for
                  improved hit testing that also facilitates accessibility
                  Sent by: public-canvas-api-request@w3.org


                  This feels like an attempt to graft retained mode
                  rendering on to canvas, what am i missing?

                  --Oliver

                  On Mar 30, 2011, at 8:39 AM, Richard Schwerdtfeger wrote:

                        Thanks Boris. Sorry for being a pest but I really
                        want developers to work through the issues.

                        So, you would do the following:

                        - Assign the a closed draw path to an element in
                        fallback content: setClickableRegion(element). This
                        immediately makes the association and places it at
                        the bottom of the list.
                        - Any time you draw the element it moves it to the
                        top of the list.
                        - If the fallback element is removed the
                        association would need to go away. I did not
                        address that so I will need to add that to the
                        proposal
                        - When a pointer event (click, etc.) goes to the
                        fallback element the normal capture/bubbling event
                        processing would apply

                        Rich


                        Rich Schwerdtfeger
                        CTO Accessibility Software Group

                        <graycol.gif>Boris Zbarsky ---03/30/2011 10:05:48
                        AM---On 3/30/11 10:55 AM, Richard Schwerdtfeger
                        wrote: > Seeing as nobody has commented can we
                        assume tha

                        From: Boris Zbarsky <bzbarsky@mit.edu>
                        To: Richard Schwerdtfeger/Austin/IBM@IBMUS
                        Cc: public-canvas-api@w3.org, public-html@w3.org,
                        public-html-a11y@w3.org, public-html-request@w3.org
                        , chuck@jumis.com, cyns@exchange.microsoft.com,
                        david.bolter@gmail.com, faulkner.steve@gmail.com,
                        franko@microsoft.com
                        Date: 03/30/2011 10:05 AM
                        Subject: Re: feedback requested: Canvas change for
                        improved hit testing that also facilitates
                        accessibility




                        On 3/30/11 10:55 AM, Richard Schwerdtfeger wrote:
                        > Seeing as nobody has commented can we assume that
                        developers have no
                        > problem with our totally changing the canvas 2D
                        API to support clickable
                        > regions?

                        It might just mean that people don't read their
                        mailing list spam every
                        few hours.

                        >From reading over your proposal, I'm not sure I
                        follow how it would
                        behave in the face of DOM mutations (e.g. elements
                        being removed from
                        the fallback content).

                        Or put another way, once you draw, you're
                        permanently attaching some
                        element to the canvas, right? Or is that not the
                        proposal?

                        -Boris