Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility

Oliver,

GDI provides the data needed for hit testing. What API are you looking for?
GDI with hit testing?

http://www.syncfusion.com/faq/windowsforms/faq_c74c.aspx

Getting back to the retained graphics discussion. In their own way
assistive technologies like magnifiers or screen readers have their own
version of a retained model of what is on the screen. They need to know:

- What are the perceivable objects?
- Where are they so I can go to them (magnifier)?
- What is the font information?
- What is the role, state, and property information?

This is why it is important to associate ALL this information with
perceivable objects on the screen.

So, what happens (I am breaking out the cobwebs).

a device context is created.
The device context has associated: font, foreground and background color,
brush, clipping region, bounding rectangle, etc.
the device context is associated with a window handle (a window handle may
have multiple device contexts associated with drawing objects within that
area.

The OS calls into each window handle with a mouse event, like a click, and
you determine if the point is within the rectangle.

Here is an example for text:
http://msdn.microsoft.com/en-us/library/dd756613
(v=vs.85).aspx#step_1:_create_a_text_layout.


What is very important is that the bounding rectangle information needed
for hit testing is garnered from the GDI device context data structure. So,
tieing this back to canvas, we would need to:

- bind a draw path encompassing each visual object on canvas to a fallback
element in the canvas subtree. At assignment time a best fit bounding
rectangle is assigned to the bounding rectangle in the accessible object
created by the user agent for that fallback element
- Have canvas manage a hierarchy  of last drawn objects to manage the hit
test. That which is last drawn is on top in the canvas. So, if there if one
object's bounding path overlaps another it is the last one drawn on top
that receives the pointing event.
- When it is determined that the hit (say an onclick) is within a drawpath
of an object pass the event to the element in the canvas subtree. This same
element should be also be capable of receiving keyboard events for that
object. Note: in Windows you might have bound these to COM.

Hope this helps.

Rich

Rich Schwerdtfeger
CTO Accessibility Software Group



From:	Oliver Hunt <oliver@apple.com>
To:	Richard Schwerdtfeger/Austin/IBM@IBMUS
Cc:	Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com,
            cyns@exchange.microsoft.com, david.bolter@gmail.com,
            faulkner.steve@gmail.com, franko@microsoft.com,
            public-canvas-api@w3.org, public-canvas-api-request@w3.org,
            public-html@w3.org, public-html-a11y@w3.org,
            public-html-request@w3.org
Date:	03/30/2011 04:19 PM
Subject:	Re: feedback requested: Canvas change for improved hit testing
            that also      facilitates accessibility



The first result i find for hit testing in GDI is
http://msdn.microsoft.com/en-us/library/ms969920.aspx

Which implies to me that they expect a developer to do their own hit
testing.

I am surprised that GDI provides any automatic hit testing for drawing to
an HDC, but before commenting further I need to see an example of the API,
can you point me at documentation or code that uses it?

--Oliver

On Mar 30, 2011, at 1:51 PM, Richard Schwerdtfeger wrote:



      Oliver,

      GDI provides and infrastructure that supports hit testing in Windows.
      The bounding rectangle of each device context is bound to the object
      and used in hit testing along with Z-order, etc. What I am telling
      you is that many developers are using canvas beyond the limitations
      and not going to SVG because for people working in the Windows space
      Canvas is closer to what they know. I am also saying that we need a
      vehicle for supplying the bounds of a UI object for accessibility.
      This has, on Windows, been tied to this device context bounding
      rectangle.

      From an accessibility perspective telling a developer they should not
      do something in canvas easily so go use SVG does not work.

      Rich


      Rich Schwerdtfeger
      CTO Accessibility Software Group

      <graycol.gif>Oliver Hunt ---03/30/2011 02:20:27 PM---On Mar 30, 2011,
      at 12:06 PM, Richard Schwerdtfeger wrote: > Oliver,

      From: Oliver Hunt <oliver@apple.com>
      To: Richard Schwerdtfeger/Austin/IBM@IBMUS
      Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com,
      cyns@exchange.microsoft.com, david.bolter@gmail.com,
      faulkner.steve@gmail.com, franko@microsoft.com,
      public-canvas-api@w3.org, public-canvas-api-request@w3.org,
      public-html@w3.org, public-html-a11y@w3.org,
      public-html-request@w3.org
      Date: 03/30/2011 02:20 PM
      Subject: Re: feedback requested: Canvas change for improved hit
      testing that also facilitates accessibility






      On Mar 30, 2011, at 12:06 PM, Richard Schwerdtfeger wrote:

            Oliver,

            It is an attempt to simplify hit testing in canvas while at the
            same time providing a vehicle to tell an assistive technology
            the bounds of the corresponding UI object being drawn on canvas
            as represented in fallback content. Currently there is no
            mapping. These bounds are needed by screen magnifiers for
            zooming and screen readers for Braille support (earlier post).

            Now the canvas author has to manage all the hit testing. This
            is a canvas deficiency that should have been in place to start
            - so grafting seems like an inappropriate response.
            Consequently, I am also seeing canvas applications that create
            the equivalent of visio by creating separate canvas elements
            overlayed on top of another canvas to represent drawing
            objects. This is very inefficient and will get worse unless
            something is done.

      This is not a canvas deficiency -- canvas is an immediate mode
      renderer, one of the things you have to handle yourself when dealing
      with _any_ immediate mode renderer is hit detection. This is true of
      Canvas, GDI, CG, raw component painting in Java, etc, etc

      If you want hit detection to be done for you, you want a retained
      mode renderer, such as SVG.

      --Oliver



            Rich Schwerdtfeger
            CTO Accessibility Software Group

            <graycol.gif>Oliver Hunt ---03/30/2011 12:11:53 PM---This feels
            like an attempt to graft retained mode rendering on to canvas,
            what am i missing? --Olive

            From: Oliver Hunt <oliver@apple.com>
            To: Richard Schwerdtfeger/Austin/IBM@IBMUS
            Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com,
            cyns@exchange.microsoft.com, david.bolter@gmail.com,
            faulkner.steve@gmail.com, franko@microsoft.com,
            public-canvas-api@w3.org, public-html@w3.org,
            public-html-a11y@w3.org, public-html-request@w3.org
            Date: 03/30/2011 12:11 PM
            Subject: Re: feedback requested: Canvas change for improved hit
            testing that also facilitates accessibility
            Sent by: public-canvas-api-request@w3.org




            This feels like an attempt to graft retained mode rendering on
            to canvas, what am i missing?

            --Oliver

            On Mar 30, 2011, at 8:39 AM, Richard Schwerdtfeger wrote:

                  Thanks Boris. Sorry for being a pest but I really want
                  developers to work through the issues.

                  So, you would do the following:

                  - Assign the a closed draw path to an element in fallback
                  content: setClickableRegion(element). This immediately
                  makes the association and places it at the bottom of the
                  list.
                  - Any time you draw the element it moves it to the top of
                  the list.
                  - If the fallback element is removed the association
                  would need to go away. I did not address that so I will
                  need to add that to the proposal
                  - When a pointer event (click, etc.) goes to the fallback
                  element the normal capture/bubbling event processing
                  would apply

                  Rich


                  Rich Schwerdtfeger
                  CTO Accessibility Software Group

                  <graycol.gif>Boris Zbarsky ---03/30/2011 10:05:48 AM---On
                  3/30/11 10:55 AM, Richard Schwerdtfeger wrote: > Seeing
                  as nobody has commented can we assume tha

                  From: Boris Zbarsky <bzbarsky@mit.edu>
                  To: Richard Schwerdtfeger/Austin/IBM@IBMUS
                  Cc: public-canvas-api@w3.org, public-html@w3.org,
                  public-html-a11y@w3.org, public-html-request@w3.org,
                  chuck@jumis.com, cyns@exchange.microsoft.com,
                  david.bolter@gmail.com, faulkner.steve@gmail.com,
                  franko@microsoft.com
                  Date: 03/30/2011 10:05 AM
                  Subject: Re: feedback requested: Canvas change for
                  improved hit testing that also facilitates accessibility




                  On 3/30/11 10:55 AM, Richard Schwerdtfeger wrote:
                  > Seeing as nobody has commented can we assume that
                  developers have no
                  > problem with our totally changing the canvas 2D API to
                  support clickable
                  > regions?

                  It might just mean that people don't read their mailing
                  list spam every
                  few hours.

                  >From reading over your proposal, I'm not sure I follow
                  how it would
                  behave in the face of DOM mutations (e.g. elements being
                  removed from
                  the fallback content).

                  Or put another way, once you draw, you're permanently
                  attaching some
                  element to the canvas, right? Or is that not the
                  proposal?

                  -Boris