- From: Richard Schwerdtfeger <schwer@us.ibm.com>
- Date: Tue, 5 Apr 2011 10:56:10 -0500
- To: Oliver Hunt <oliver@apple.com>
- Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com, faulkner.steve@gmail.com, franko@microsoft.com, public-canvas-api@w3.org, public-canvas-api-request@w3.org, public-html@w3.org, public-html-a11y@w3.org, public-html-request@w3.org
- Message-ID: <OFD126773D.F79C391C-ON86257869.005535B3-86257869.005789E9@us.ibm.com>
Oliver, GDI provides the data needed for hit testing. What API are you looking for? GDI with hit testing? http://www.syncfusion.com/faq/windowsforms/faq_c74c.aspx Getting back to the retained graphics discussion. In their own way assistive technologies like magnifiers or screen readers have their own version of a retained model of what is on the screen. They need to know: - What are the perceivable objects? - Where are they so I can go to them (magnifier)? - What is the font information? - What is the role, state, and property information? This is why it is important to associate ALL this information with perceivable objects on the screen. So, what happens (I am breaking out the cobwebs). a device context is created. The device context has associated: font, foreground and background color, brush, clipping region, bounding rectangle, etc. the device context is associated with a window handle (a window handle may have multiple device contexts associated with drawing objects within that area. The OS calls into each window handle with a mouse event, like a click, and you determine if the point is within the rectangle. Here is an example for text: http://msdn.microsoft.com/en-us/library/dd756613 (v=vs.85).aspx#step_1:_create_a_text_layout. What is very important is that the bounding rectangle information needed for hit testing is garnered from the GDI device context data structure. So, tieing this back to canvas, we would need to: - bind a draw path encompassing each visual object on canvas to a fallback element in the canvas subtree. At assignment time a best fit bounding rectangle is assigned to the bounding rectangle in the accessible object created by the user agent for that fallback element - Have canvas manage a hierarchy of last drawn objects to manage the hit test. That which is last drawn is on top in the canvas. So, if there if one object's bounding path overlaps another it is the last one drawn on top that receives the pointing event. - When it is determined that the hit (say an onclick) is within a drawpath of an object pass the event to the element in the canvas subtree. This same element should be also be capable of receiving keyboard events for that object. Note: in Windows you might have bound these to COM. Hope this helps. Rich Rich Schwerdtfeger CTO Accessibility Software Group From: Oliver Hunt <oliver@apple.com> To: Richard Schwerdtfeger/Austin/IBM@IBMUS Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com, faulkner.steve@gmail.com, franko@microsoft.com, public-canvas-api@w3.org, public-canvas-api-request@w3.org, public-html@w3.org, public-html-a11y@w3.org, public-html-request@w3.org Date: 03/30/2011 04:19 PM Subject: Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility The first result i find for hit testing in GDI is http://msdn.microsoft.com/en-us/library/ms969920.aspx Which implies to me that they expect a developer to do their own hit testing. I am surprised that GDI provides any automatic hit testing for drawing to an HDC, but before commenting further I need to see an example of the API, can you point me at documentation or code that uses it? --Oliver On Mar 30, 2011, at 1:51 PM, Richard Schwerdtfeger wrote: Oliver, GDI provides and infrastructure that supports hit testing in Windows. The bounding rectangle of each device context is bound to the object and used in hit testing along with Z-order, etc. What I am telling you is that many developers are using canvas beyond the limitations and not going to SVG because for people working in the Windows space Canvas is closer to what they know. I am also saying that we need a vehicle for supplying the bounds of a UI object for accessibility. This has, on Windows, been tied to this device context bounding rectangle. From an accessibility perspective telling a developer they should not do something in canvas easily so go use SVG does not work. Rich Rich Schwerdtfeger CTO Accessibility Software Group <graycol.gif>Oliver Hunt ---03/30/2011 02:20:27 PM---On Mar 30, 2011, at 12:06 PM, Richard Schwerdtfeger wrote: > Oliver, From: Oliver Hunt <oliver@apple.com> To: Richard Schwerdtfeger/Austin/IBM@IBMUS Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com, faulkner.steve@gmail.com, franko@microsoft.com, public-canvas-api@w3.org, public-canvas-api-request@w3.org, public-html@w3.org, public-html-a11y@w3.org, public-html-request@w3.org Date: 03/30/2011 02:20 PM Subject: Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility On Mar 30, 2011, at 12:06 PM, Richard Schwerdtfeger wrote: Oliver, It is an attempt to simplify hit testing in canvas while at the same time providing a vehicle to tell an assistive technology the bounds of the corresponding UI object being drawn on canvas as represented in fallback content. Currently there is no mapping. These bounds are needed by screen magnifiers for zooming and screen readers for Braille support (earlier post). Now the canvas author has to manage all the hit testing. This is a canvas deficiency that should have been in place to start - so grafting seems like an inappropriate response. Consequently, I am also seeing canvas applications that create the equivalent of visio by creating separate canvas elements overlayed on top of another canvas to represent drawing objects. This is very inefficient and will get worse unless something is done. This is not a canvas deficiency -- canvas is an immediate mode renderer, one of the things you have to handle yourself when dealing with _any_ immediate mode renderer is hit detection. This is true of Canvas, GDI, CG, raw component painting in Java, etc, etc If you want hit detection to be done for you, you want a retained mode renderer, such as SVG. --Oliver Rich Schwerdtfeger CTO Accessibility Software Group <graycol.gif>Oliver Hunt ---03/30/2011 12:11:53 PM---This feels like an attempt to graft retained mode rendering on to canvas, what am i missing? --Olive From: Oliver Hunt <oliver@apple.com> To: Richard Schwerdtfeger/Austin/IBM@IBMUS Cc: Boris Zbarsky <bzbarsky@mit.edu>, chuck@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com, faulkner.steve@gmail.com, franko@microsoft.com, public-canvas-api@w3.org, public-html@w3.org, public-html-a11y@w3.org, public-html-request@w3.org Date: 03/30/2011 12:11 PM Subject: Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility Sent by: public-canvas-api-request@w3.org This feels like an attempt to graft retained mode rendering on to canvas, what am i missing? --Oliver On Mar 30, 2011, at 8:39 AM, Richard Schwerdtfeger wrote: Thanks Boris. Sorry for being a pest but I really want developers to work through the issues. So, you would do the following: - Assign the a closed draw path to an element in fallback content: setClickableRegion(element). This immediately makes the association and places it at the bottom of the list. - Any time you draw the element it moves it to the top of the list. - If the fallback element is removed the association would need to go away. I did not address that so I will need to add that to the proposal - When a pointer event (click, etc.) goes to the fallback element the normal capture/bubbling event processing would apply Rich Rich Schwerdtfeger CTO Accessibility Software Group <graycol.gif>Boris Zbarsky ---03/30/2011 10:05:48 AM---On 3/30/11 10:55 AM, Richard Schwerdtfeger wrote: > Seeing as nobody has commented can we assume tha From: Boris Zbarsky <bzbarsky@mit.edu> To: Richard Schwerdtfeger/Austin/IBM@IBMUS Cc: public-canvas-api@w3.org, public-html@w3.org, public-html-a11y@w3.org, public-html-request@w3.org, chuck@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com, faulkner.steve@gmail.com, franko@microsoft.com Date: 03/30/2011 10:05 AM Subject: Re: feedback requested: Canvas change for improved hit testing that also facilitates accessibility On 3/30/11 10:55 AM, Richard Schwerdtfeger wrote: > Seeing as nobody has commented can we assume that developers have no > problem with our totally changing the canvas 2D API to support clickable > regions? It might just mean that people don't read their mailing list spam every few hours. >From reading over your proposal, I'm not sure I follow how it would behave in the face of DOM mutations (e.g. elements being removed from the fallback content). Or put another way, once you draw, you're permanently attaching some element to the canvas, right? Or is that not the proposal? -Boris
Attachments
- image/gif attachment: graycol.gif
Received on Tuesday, 5 April 2011 15:57:10 UTC