Re: hit testing and retained graphics

Paul,

The proposal is intended for vector graphics managed by canvas. I understand that your development team simply uses PNG files an associates them for hit testing; you've proposed CSS attributes be applied so that the transparent sections of such images pass thru pointer events.

It's difficult to make those objects 'accessible' beyond simply sharing the bounding box of the png with the Accessibility API.


-Charles


On Jun 22, 2011, at 5:25 AM, Paul Bakaus <pbakaus@zynga.com> wrote:

> Hi Charles,
> 
> I've been thinking about this but it won't do what I need. It adds a lot
> of additional DOM elements and it's still not a put-on-and-forget-about-it
> solution. You would have to pregenerate geometric shapes as developer. Not
> necessarily a fun thing to do for most web devs.
> 
> Cheers,
> Paul
> 
> Am 22.06.11 14:16 schrieb "Charles McCathieNevile" unter
> <chaals@opera.com>:
> 
>> On Wed, 22 Jun 2011 13:06:08 +0200, Paul Bakaus <pbakaus@zynga.com> wrote:
>> 
>>> Hi everyone,
>>> 
>>> We at Zynga use hit testing to a great extend in our isometric games and
>>> have long searched for a solution that removes some of the processing
>>> burden from us. I therefore very much disagree that hit testing should
>>> be up to the js developer. If possible, this is a perfect candidate for
>>> a job the browser can help us with. And by "us", I am probably talking
>>> about almost every JS game developer.
>> 
>> What about placing a transparent image over the top of your canvas, and
>> using an image map in it. Things that need to be clickable can be updated
>> according to whatever you're drawing, things that don't remain purely
>> immediate mode.
>> 
>> I haven't tried it, but I've been thinking about it (and about how the
>> whole discussion reminds me of ISSUE-105 [1]) and I ran across something
>> that demonstrates the technique [2] (although I think the purpose of that
>> demo was something else, and whatever it was trying to do seems horribly
>> complex to me).
>> 
>> [1] http://www.w3.org/html/wg/tracker/issues/105
>> [2] http://zreference.com/image-map-canvas/
>> 
>> cheers
>> 
>> Chaals
>> 
>>> I proposed an extension to CSS in April that a lot of folks I talked to
>>> liked, as it is both simple and effective for simple hit testing on DOM
>>> elements. You can read the summary in this thread:
>>> http://lists.w3.org/Archives/Public/www-style/2011Apr/0558.html
>>> 
>>> Paul
>>> 
>>> Von: Charles Pritchard <chuck@jumis.com<mailto:chuck@jumis.com>>
>>> Datum: Tue, 21 Jun 2011 21:36:57 -0700
>>> An: Richard Schwerdtfeger <schwer@us.ibm.com<mailto:schwer@us.ibm.com>>
>>> Cc: Frank Olivier
>>> <Frank.Olivier@microsoft.com<mailto:Frank.Olivier@microsoft.com>>,
>>> Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>,
>>> "david.bolter@gmail.com<mailto:david.bolter@gmail.com>"
>>> <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>,
>>> "Mike@w3.org<mailto:Mike@w3.org>" <Mike@w3.org<mailto:Mike@w3.org>>,
>>> "public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>"
>>> <public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>>,
>>> "public-html@w3.org<mailto:public-html@w3.org>"
>>> <public-html@w3.org<mailto:public-html@w3.org>>,
>>> "public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>"
>>> <public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>>,
>>> "public-html-request@w3.org<mailto:public-html-request@w3.org>"
>>> <public-html-request@w3.org<mailto:public-html-request@w3.org>>
>>> Betreff: Re: hit testing and retained graphics
>>> 
>>> This is a lengthy e-mail. Summary; three parts:
>>> 
>>> 1. Retained Path objects are ubiquitous in basic 2d drawing APIs;
>>> they are efficient, they are managed by the developer, and
>>> they are necessary for accessibility.
>>> 
>>> 2. Basic additions to the canvas spec can introduce path objects
>>> which can be bound to DOM elements.
>>> 
>>> 3. An example of three buttons in a canvas element is shown,
>>> demonstrating the API.
>>> 
>>> 4. Some usage notes.
>>> 
>>> ....................
>>> 
>>> Part 1. Path objects are common in drawing APIs.
>>> 
>>> I've worked a little with Richard's proposal, and done a quick review on
>>> the major drawing APIs out there that
>>> work on the same level of abstraction as Canvas.
>>> 
>>> 
>>> http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/
>>> display/GraphicsPath.html
>>> 
>>> http://msdn.microsoft.com/en-us/library/system.drawing.drawing2d.pathdata
>>> _members(v=vs.85).aspx
>>> 
>>> http://download.oracle.com/javase/7/docs/api/java/awt/geom/PathIterator.h
>>> tml
>>> 
>>> http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Co
>>> coaDrawingGuide/Paths/Paths.html
>>> "If you draw the same content repeatedly ... [it] is usually more
>>> efficient to retain an existing NSBezierPath object than to recreate it
>>> during each drawing cycle."
>>> 
>>> That last excerpt applies to a few rendering APIs, not just Apple's
>>> Cocoa.
>>> There's precedent in each of these APIs for a Path data object.
>>> WebGL uses path data by typed arrays. It does not include curves.
>>> SVG has normalized path types:
>>> http://msdn.microsoft.com/en-us/library/ff971968(v=vs.85).aspx
>>> 
>>> I do not think we need "retained graphics" in canvas -- but it'd be a
>>> huge boost to developers,
>>> and to accessibility, if we can have a means for sharing path data
>>> between layers.
>>> The HTML map tag is insufficient... We're looking for a replacement.
>>> 
>>> Richard has put forward a few proposals, as have I... If we can get
>>> some traction with the group, I'm sure we can add a great feature
>>> to Canvas, making life easier on developers and AT vendors alike.
>>> 
>>> Developers have told me that they want mouse events delegated
>>> to the shadow dom, based on paths they've bound. Richard and I
>>> are most concerned with getting path data to ATs; though Richard
>>> has a warmth in his heart for making things easier on developers.
>>> 
>>> Many vendors are concerned about mixing retained-mode concepts
>>> with Canvas's immediate mode API.
>>> I hope that by focusing only on static path data, we can move past
>>> that concern. I also hope that by introducing new methods, we can
>>> see improved efficiency and accessibility in Canvas applications.
>>> 
>>> Please take the following in good faith, and send the list feedback,
>>> so we can take that feedback, apply it, and push the proposal into
>>> the view of a wider audience.
>>> 
>>> Part 2. Creating a CanvasPath object and stringify methods in the spec.
>>> 
>>> I've done a basic supplemental. setElementFromPath binds the current
>>> path onto an element, with an optional zIndex. I've also done a
>>> stringify,
>>> supplemental, which would make life a lot easier on developers when it
>>> comes to getting a string for the current path or transformation matrix.
>>> 
>>> // CanvasPath extension
>>> [Supplemental]
>>> interface CanvasRenderingContext2D {
>>>    void beginFromPath(in CanvasPath path) ;
>>>    void setElementPath(in Element element, in optional DOMString
>>> zIndex) ;
>>>    CanvasPath getCurrentPath();
>>> }
>>> 
>>> interface CanvasPath {
>>>    // opaque object
>>> }
>>> 
>>> // Stringify extensions, defined by SVG normalized paths and CSSMatrix.
>>> [Supplemental]
>>> interface CanvasRenderingContext2D {
>>>    void beginFromPath(in normalizedPathSegList path) ;
>>>    void beginFromPath(in DOMString path) ;
>>>    CSSMatrix getCurrentMatrix();
>>>    void setCurrentMatrix(in CSSMatrix matrix);
>>>    void setCurrentMatrix(in DOMString matrix);
>>> }
>>> [Supplemental]
>>> interface CanvasPath {
>>>    stringifier; // returns normalized path
>>> }
>>> 
>>> 
>>> Part 3. Example of an accessible Canvas, keyboard and pointer device.
>>> /*
>>> Draw three buttons, 100x100 pixels each, spaced 40px apart
>>>    If a button is in focus, set its zIndex higher than the other
>>> buttons,
>>>    in case of overlap if spacing is changed.
>>> */
>>> 
>>> var canvas = document.getElementById(ŒcanvasTest¹); canvas.width += 0;
>>> // reset state.
>>> var ctx = canvas.getContext(¹2d¹);
>>> var buttons = { 'a': 'Button A', 'b': 'Button B','c': 'Button C' }
>>> var width = 100; var height=30; var padding = 40;
>>> ctx.font = '20px sans-serif';
>>> ctx.beginPath();
>>> ctx.rect(0,0,width,height);
>>> var path = ctx.getCurrentPath();
>>> for(var i in buttons) {
>>>    var button = document.getElementById('button_'+i);
>>>    if(!button) {
>>>        button = document.createElement('input');
>>>        button.setAttribute('id', 'button_'+i);
>>>        button.setAttribute('name', i);
>>>        button.setAttribute('value', buttons[i]);
>>>        button.setAttribute('type','submit');
>>>        if(0) button.onclick = function() { alert("It works!"); };
>>>        if(0) button.onfocus = function() {
>>>            if(button.getAttribute('data-myeasypathdata')) {
>>>                button.parentNode.getContext('2d').beginFromPath(
>>> button.getAttribute('data-myeasypathdata') );
>>> 
>>> button.parentNode.getContext('2d').drawFocusRing(button);
>>>            }
>>>        };
>>>        canvas.appendChild(button);
>>>    }
>>>    var isFocused = document.activeElement == button;
>>>    var zIndex = isFocused ? "1" : "0";
>>>    ctx.beginFromPath(path);
>>>    ctx.setElementPath(button, zIndex);
>>>    if(isFocused) ctx.drawFocusRing(button);
>>>    if(0) button.setAttribute('data-myeasypathdata',
>>> ctx.getCurrentPath());
>>>    ctx.fillStyle = 'black';
>>>    ctx.fill();
>>>    ctx.fillStyle = 'white';
>>>    var textBaseline = ....;
>>>    ctx.fillText(buttons[i], ( width - ctx.measureText(buttons[i]).width
>>> ) / 2, textBaseline);
>>>    ctx.translate(width + padding, 0);
>>> }
>>> 
>>> 
>>> Part 4. Usage notes.
>>> 
>>> Stringify options allow developers more room for loose coupling of
>>> events;
>>> they also allow better interop between SVG and Canvas... They help with
>>> debugging;
>>> they can reduce the overhead of complex objects, by reducing the number
>>> of method
>>> calls, considerably.
>>> 
>>> z-index is necessary for coordinate/pointer-based interfaces; things
>>> overlap.
>>> 
>>> Transformations will be used heavily in animations. Transformations are
>>> live, in canvas.
>>> Notice that only one translate method is called, incrementally moving
>>> the origin along the x axis.
>>> 
>>> 
>>> -----
>>> 
>>> 
>>> There are several ATs which require coordinates / paths ahead of time.
>>> Some ATs get by this issue by focusing on an element before requesting
>>> the bounding box.
>>> 
>>> ViewPlus produces the "IVEO Hands-on-Learning System":
>>> http://www.viewplus.com/
>>> 
>>> Apple produces Mobile Safari, with "VoiceOver":
>>> http://www.apple.com/accessibility/iphone/vision.html
>>> 
>>> Both of these products weighed heavily in the design of the proposal.
>>> 
>>> Input is appreciated.
>>> 
>>> -Charles
>>> 
>>> 
>>> On 6/20/2011 3:45 PM, Richard Schwerdtfeger wrote:
>>> 
>>> I have some concerns about this. It means:
>>> 
>>> - The author has to do the transformations
>>> - We have to add DOM attributes
>>> - The mouse events are routed to canvas and not the sub-DOM elements
>>> where the keyboard handling is going on. The user agent could process
>>> the mouse events at canvas and then propagate them to the corresponding
>>> DOM object in the subtree.
>>> 
>>> Your proposal does have the advantage that the bounds of the objects are
>>> in the DOM but for HTML we don't have this for any of the DOM elements
>>> now.
>>> 
>>> What I was thinking was the following:
>>> 
>>> - Today Canvas has the notion of a context
>>> - We allow the author to have the same context (with methods) for each
>>> drawing object and apply a bounds and z-index as you suggest.
>>> - We then bind each drawing object to the canvas subtree DOM element:
>>> 
>>> So each drawing object would be an instance of a canvas context with
>>> methods were we do something like:
>>> 
>>> 1. we assume that the canvas element when the page is created is an
>>> instance of a canvasObject (having a context)
>>> 2. we assume that drawingOjects are a subclass of canvasObject that
>>> support all the canvas2DAPI in canvasObject with some additions such as:
>>> - ZIndex attribute
>>> - a bounding drawing path and methods for modifying them
>>> - a method for associating the drawingObject with a canvas subtree DOM
>>> element.
>>> 3. we add an method to canvas that says addDrawObject.
>>> On the canvas element we have the following:
>>> 
>>> var canvas = document.getElementById(ŒcanvasTest¹);
>>> if (canvas.getContext){
>>> var ctx = canvas.getContext(¹2d¹);
>>> DO = new drawingObject();
>>> dctx= DO.getContext('2d');
>>> dctx.ZIndex="2";
>>> dctx.beginPath();
>>> dctx.moveTo(130,100);
>>> dctx.lineTo...
>>> ...
>>> dctx.setPathtoBounds();
>>> dctx.setDOMSubtreeNode(foo); //Foo is a subtree node where keyboard
>>> events go to and we do our accessibility enablement to populate the
>>> accessibility tree
>>> //Internally the user agents maps the bounding rectangle for the
>>> accessible of the DOM object as a best fit rectangle of the
>>> //path used to form the bounds of the drawing object
>>> ctx.addDrawingObject(DO);
>>> }
>>> 
>>> Now the user agent the information needed to do the hit testing and
>>> retained mode graphics (I am oversimplifying for illustration purposes)
>>> to be able to track the pointing device input and routing it to the same
>>> DOM objects that process the keyboard and all the other accessibility
>>> information. This includes hit testing.
>>> 
>>> Mike provided feedback on the HTML A11Y call that authors did not want
>>> to do the hit testing themselves.
>>> Rich Schwerdtfeger
>>> CTO Accessibility Software Group
>>> 
>>> [cid:part1.01000009.05010600@jumis.com]Frank Olivier ---06/20/2011
>>> 11:01:54 AM---I would leave hit testing up to the (javascript) author. I
>>> would recommend that they set existing x,
>>> 
>>> From: Frank Olivier
>>> <Frank.Olivier@microsoft.com><mailto:Frank.Olivier@microsoft.com>
>>> To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
>>> "chuck@jumis.com"<mailto:chuck@jumis.com>
>>> <chuck@jumis.com><mailto:chuck@jumis.com>,
>>> "Mike@w3.org"<mailto:Mike@w3.org> <Mike@w3.org><mailto:Mike@w3.org>,
>>> "david.bolter@gmail.com"<mailto:david.bolter@gmail.com>
>>> <david.bolter@gmail.com><mailto:david.bolter@gmail.com>, Cynthia Shelly
>>> <cyns@microsoft.com><mailto:cyns@microsoft.com>
>>> Cc: "public-canvas-api@w3.org"<mailto:public-canvas-api@w3.org>
>>> <public-canvas-api@w3.org><mailto:public-canvas-api@w3.org>,
>>> "public-html-a11y@w3.org"<mailto:public-html-a11y@w3.org>
>>> <public-html-a11y@w3.org><mailto:public-html-a11y@w3.org>,
>>> "public-html@w3.org"<mailto:public-html@w3.org>
>>> <public-html@w3.org><mailto:public-html@w3.org>
>>> Date: 06/20/2011 11:01 AM
>>> Subject: RE: hit testing and retained graphics
>>> Sent by: public-html-request@w3.org<mailto:public-html-request@w3.org>
>>> 
>>> ________________________________
>>> 
>>> 
>>> 
>>> I would leave hit testing up to the (javascript) author. I would
>>> recommend that they set existing x,y position, z-index attributes on the
>>> DOM objects in the canvas subtree to report what the UI 'looks like' to
>>> AT tools. This way, the AT tools don't need to change - this part of the
>>> DOM is no different to them than any other part - and authors need to be
>>> annotating canvas DOM objects with correct information anyway (labels,
>>> aria attributes, etc).
>>> 
>>> From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
>>> Sent: Friday, June 17, 2011 11:42 AM
>>> To: chuck@jumis.com<mailto:chuck@jumis.com>; Frank Olivier;
>>> Mike@w3.org<mailto:Mike@w3.org>;
>>> david.bolter@gmail.com<mailto:david.bolter@gmail.com>; Cynthia Shelly
>>> Cc: public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>;
>>> public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>;
>>> public-html@w3.org<mailto:public-html@w3.org>
>>> Subject: hit testing and retained graphics
>>> 
>>> Charles, Frank, Mike,
>>> 
>>> I am back from vacation. How far do we need to go with hit testing?
>>> Right now I am looking at associating a closed draw path with a DOM
>>> object in the canvas subtree. We would then need to address the routing
>>> of pointing device input events to the DOM object. The drawing path can
>>> be used to provide bound information to platform accessibility API.
>>> 
>>> Do we need to bind any other drawing properties to the canvas object -
>>> similar to the way device context's are handled on graphic subsystems
>>> like Windows?
>>> 
>>> Mike, I am including you as before I went on vacation you indicated that
>>> a number of developers desired this feature and wanted to be involved.
>>> 
>>> Rich
>>> 
>>> 
>>> Rich Schwerdtfeger
>>> CTO Accessibility Software Group
>>> 
>>> 
>>> 
>> 
>> 
>> --
>> Charles McCathieNevile  Opera Software, Standards Group
>>    je parle français -- hablo español -- jeg lærer norsk
>> http://my.opera.com/chaals       Try Opera: http://www.opera.com
>> 
> 
>

Received on Wednesday, 22 June 2011 17:54:08 UTC