RE: Request to re-open issue 131 -USE CASES, USE CASES, USE CASES

I've created a more normative change at http://www.w3.org/wiki/Canvas_hit_testing. I think the actual spec change that is required for this is relatively small, as the underlying concept (associate a path; forward all events) is simple. Please let me know if you have any questions or feedback.

Thanks
Frank

From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
Sent: Monday, January 09, 2012 10:50 AM
To: Steve Faulkner
Cc: chuck@jumis.com; Cynthia Shelly; david bolter; David Bolter; Frank Olivier; Jonas Sicking; Maciej Stachowiak; Paul Cotton; public-canvas-api@w3.org; public-html@w3.org; public-html-a11y@w3.org; Sam Ruby
Subject: Re: Request to re-open issue 131 -USE CASES, USE CASES, USE CASES


We need a normative text change to review.


Rich Schwerdtfeger

[Inactive hide details for Steve Faulkner ---01/09/2012 05:36:00 AM---Hi Rich, Frank wrote provided info here: http://www.w3.org]Steve Faulkner ---01/09/2012 05:36:00 AM---Hi Rich, Frank wrote provided info here: http://www.w3.org/wiki/Canvas_hit_testing

From: Steve Faulkner <faulkner.steve@gmail.com<mailto:faulkner.steve@gmail.com>>
To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
Cc: David Bolter <dbolter@mozilla.com<mailto:dbolter@mozilla.com>>, chuck@jumis.com<mailto:chuck@jumis.com>, Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>, david bolter <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>, franko@microsoft.com<mailto:franko@microsoft.com>, Jonas Sicking <jonas@sicking.cc<mailto:jonas@sicking.cc>>, Maciej Stachowiak <mjs@apple.com<mailto:mjs@apple.com>>, Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>, public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>, public-html@w3.org<mailto:public-html@w3.org>, public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>, Sam Ruby <rubys@intertwingly.net<mailto:rubys@intertwingly.net>>
Date: 01/09/2012 05:36 AM
Subject: Re: Request to re-open issue 131 -USE CASES, USE CASES, USE CASES

________________________________



Hi Rich,

Frank wrote provided info here: http://www.w3.org/wiki/Canvas_hit_testing

Is that enough to work with or do we need frank to provide more details?

regards
steve

On 8 January 2012 17:57, Richard Schwerdtfeger <schwer@us.ibm.com<mailto:schwer@us.ibm.com>> wrote:
We just need Frank to write up a formal change proposal. Apparently, we are now creating a new document for accessibility extensions to Canvas 2D API for which Steve and I will maintain. As we gain implementations the chairs can decide how best to integrate the implemented work back in.


Rich Schwerdtfeger

[Inactive hide details for David Bolter ---12/20/2011 10:19:28 AM---Hi all, I think Frank's proposal is reasonable and I haven't]David Bolter ---12/20/2011 10:19:28 AM---Hi all, I think Frank's proposal is reasonable and I haven't come up with a better idea. It think it

From: David Bolter <dbolter@mozilla.com<mailto:dbolter@mozilla.com>>

To: Steve Faulkner <faulkner.steve@gmail.com<mailto:faulkner.steve@gmail.com>>,
Cc: Richard Schwerdtfeger/Austin/IBM@IBMUS, chuck@jumis.com<mailto:chuck@jumis.com>, Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>, david bolter <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>, franko@microsoft.com<mailto:franko@microsoft.com>, Maciej Stachowiak <mjs@apple.com<mailto:mjs@apple.com>>, Paul Cotton <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>, public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>, public-html@w3.org<mailto:public-html@w3.org>, public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>, Sam Ruby <rubys@intertwingly.net<mailto:rubys@intertwingly.net>>, Jonas Sicking <jonas@sicking.cc<mailto:jonas@sicking.cc>>
Date: 12/20/2011 10:19 AM
Subject: Re: Request to re-open issue 131 -USE CASES, USE CASES, USE CASES

________________________________




Hi all,

I think Frank's proposal is reasonable and I haven't come up with a better idea. It think it is definitely worth a thoughtful and careful read.

Note I recall Paul Bakaus (from Zynga) was following the hit testing discussion in the Summer. Paul I'd be curious to hear your fresh thoughts on what you think of Frank's proposal. I'm also curious this approach would make the canvas sub-dom more attractive to you guys.

Cheers,
David


----- Original Message -----
> From: "Steve Faulkner" <faulkner.steve@gmail.com<mailto:faulkner.steve@gmail.com>>
> To: "Jonas Sicking" <jonas@sicking.cc<mailto:jonas@sicking.cc>>
> Cc: "Richard Schwerdtfeger" <schwer@us.ibm.com<mailto:schwer@us.ibm.com>>, chuck@jumis.com<mailto:chuck@jumis.com>, "Cynthia Shelly" <cyns@microsoft.com<mailto:cyns@microsoft.com>>, "david
> bolter" <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>, dbolter@mozilla.com<mailto:dbolter@mozilla.com>, franko@microsoft.com<mailto:franko@microsoft.com>, "Maciej Stachowiak" <mjs@apple.com<mailto:mjs@apple.com>>,
> "Paul Cotton" <Paul.Cotton@microsoft.com<mailto:Paul.Cotton@microsoft.com>>, public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>, public-html@w3.org<mailto:public-html@w3.org>, public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>, "Sam
> Ruby" <rubys@intertwingly.net<mailto:rubys@intertwingly.net>>
> Sent: Tuesday, December 20, 2011 5:35:00 AM
> Subject: Re: Fw: Request to re-open issue 131 -USE CASES, USE CASES, USE CASES
> Hi Jonas,
>
> you wrote:
>
> > I honestly have lost track of what the latest proposal is at this
> > point.
>
> Frank's proposal:
>
> http://www.w3.org/wiki/Canvas_hit_testing
>
> here it is inline:
>
> IMO We need a 'general purpose' hit testing solution here (to assist
> in author uptake) with a very simple method that allows authors to see
> what path/pixels are actually being set for hit testing:
>
> boolean setElementPath(in Element element);
>
> I would define this as: (Additional spec text for
> http://dev.w3.org/html5/canvas-extensions/Overview.html#focus-management-1)
>
> When a canvas is interactive, authors should include focusable
> elements in the element's fallback content corresponding to each
> focusable part of the canvas.
>
> When multiple focusable elements are added, authors should use
> setElementPath() to set the focus ring of each individual focusable
> element. If the focus ring is not set with setElementPath(), the focus
> ring of a focusable element in the fallback content is the bounding
> rectangle of the parent canvas element. [This improves accessibility
> for the case where the entire canvas element represents a single
> interactive control (think very simple custom-drawn checkbox), and
> fallback element click handling is being handled entirely by the
> author. [The single checkbox case.]]
>
> When setElementPath() is called, the drawing path is used to form the
> focus ring provided that drawing path contains a closed path. The
> drawing path is used to form a best fit bounding rectangle in screen
> coordinates. The bounding rectangle and drawing path may be used to
> enhance accessibility properties [ARIA] for the targeted element.
>
> User agents should use the information set by setElementPath() to
> create accessible user experiences. For example, a screen reader may
> read the fallback element's details when the user indicates interest
> in that region of the canvas.
>
> The setElementPath(element) method, when invoked, must run the
> following steps: 1. If the element is not a descendant of the canvas
> element with whose context the method is associated, then return false
> and abort these steps.
>
> 2. If supporting an accessibility API, user agents may use the drawing
> path to form a best fit rectangle in screen coordinates and apply it
> to the bounding rectangle of the associated accessible object. The
> focus ring should be subject to the clipping region.
>
> 3. Return true.
>
> When the user interacts with the canvas, the user agent should forward
> the event to the fallback element.
>
> If two or more elements have overlapping paths (set via
> setElementPath()) the last call to setElementPath() wins.
>
> regards
> Stevef
>
> On 20 December 2011 10:22, Jonas Sicking <jonas@sicking.cc<mailto:jonas@sicking.cc>> wrote:
> > On Mon, Dec 19, 2011 at 1:19 PM, Richard Schwerdtfeger
> > <schwer@us.ibm.com<mailto:schwer@us.ibm.com>>
> > wrote:
> >>
> >> Jonas,
> >>
> >> For purposes of agreement getting some consensus I would like to
> >> put the
> >> text discussion and focus on this use case which you had agreed we
> >> should
> >> support while at TPAC:
> >>
> >>
> >>
> >> 1. Hit Testing and the bounds of an object
> >>
> >> USE CASE: Regarding hit testing, it is very, very simple. In ALL
> >> operating
> >> systems that support an accessibility API it is ESSENTIAL that a
> >> magnifier
> >> be able to determine the location of an accessible object on the
> >> screen so
> >> that a user may zoom to it. It has absolutely nothing to do with
> >> rich text
> >> editing other than the fact that like all other objects we would
> >> need to
> >> find the text box to zoom to it. You and I, who can see, can scan a
> >> page and
> >> find what we want. Yet, a magnifier user may only be able to see,
> >> say a text
> >> box, which has focus and a few characters as the screen my be
> >> magnified by a
> >> factor of 10. The few characters in the text box may be all they
> >> see on the
> >> screen. So, to zoom to something else they will ask their assistive
> >> technology to do things like find an object and zoom to it - or
> >> they may ask
> >> it to read from the beginning of an application at the first
> >> accessible
> >> object and maintain a magnification point around the object
> >>
> >> Unlike HTML accessible canvas object reside in fallback content
> >> which is
> >> NOT visible. So, the screen location of these objects can NOT be
> >> found
> >> without programmatic intervention. In ALL accessible GUI OS
> >> platforms the
> >> bound so the drawing object are acquired from the device context
> >> which is
> >> mapped ultimately to the drawing object and then to the
> >> corresponding
> >> accessible object. The screen location is typically the same
> >> location used
> >> in hit testing.
> >>
> >> USE CASE: USE Braille devices also use the bounding information to
> >> assist
> >> in line breaks on Braille displays.
> >>
> >> How do I know these things? I built the offscreen model for the
> >> first GUI
> >> screen readers for the PC. I was hip deep in the graphics engine
> >> and
> >> windowing systems for both OS/2 and Windows. I also worked on one
> >> of the
> >> first screen magnifiers the PC - Screen Magnifier/2.
> >>
> >> So, there are your use cases. There is NO invention here and the
> >> text
> >> editor case is really a red herring as it is not the essential
> >> reason why we
> >> need the bounds and hit testing.
> >>
> >> USE CASE: The use case for hit testing is it pushes the load off
> >> the
> >> author to the user agent. Imagine you having to do all the GUI hit
> >> testing
> >> manually for your Windows app. Also, now, pointing device handling
> >> occurs at
> >> the canvas element while the keyboard handling is handled at an
> >> element in
> >> fallback content.
> >>
> >> Here is the accessibility API for UNIX Systems that needs the
> >> bounds (see
> >> BoundingBox) of an object:
> >> http://people.gnome.org/~billh/at-spi-idl/html/classAccessibility_1_1Component.html<http://people.gnome.org/%7Ebillh/at-spi-idl/html/classAccessibility_1_1Component.html>
> >> Here is the accessibility API (see accLocation) for MSAA which is
> >> used
> >> both Chrome and Firefox on Windows:
> >> http://msdn.microsoft.com/en-us/library/dd318466.aspx
> >> Here it the accessibility API (see Bounding Box) for an UIA
> >> provider:
> >> http://msdn.microsoft.com/en-us/library/ms726714(v=VS.85).aspx<http://msdn.microsoft.com/en-us/library/ms726714%28v=VS.85%29.aspx>
> >>
> >> Right now, without a change to canvas we cannot supply this
> >> information to
> >> assistive technologies.
> >
> >
> > Yes, I definitely support the ability to associate an area of the
> > canvas
> > with a element in the sub-dom (sorry, forget what the official name
> > is, if
> > there is one) of the canvas element. This will enable things like
> > hit-testing, driving screen magnifiers, implementing
> > scrolling-to-part-of-canvas, etc.
> >
> > I apologize if I gave the impression of otherwise.
> >
> >>
> >> Do you support Frank moving forward with the setElementPath/hit
> >> test
> >> proposal for the working group to review and are you still
> >> supportive of
> >> having such an API for canvas?
> >
> >
> > I honestly have lost track of what the latest proposal is at this
> > point. The
> > main goal I have is to create an API which is simple enough to use
> > for
> > people to want to do their own canvas hit-testing using the API we
> > provide.
> > That is how we can get the most number of people to use these APIs,
> > and thus
> > create the most accessible web.
> >
> > / Jonas




--
with regards

Steve Faulkner
Technical Director - TPG

www.paciellogroup.com<http://www.paciellogroup.com/> | www.HTML5accessibility.com<http://www.html5accessibility.com/> | www.twitter.com/stevefaulkner<http://www.twitter.com/stevefaulkner>
HTML5: Techniques for providing useful text alternatives - dev.w3.org/html5/alt-techniques/<http://dev.w3.org/html5/alt-techniques/>
Web Accessibility Toolbar - www.paciellogroup.com/resources/wat-ie-about.html <http://www.paciellogroup.com/resources/wat-ie-about.html>