Re: hit testing and retained graphics

Hi Charles,

I've been thinking about this but it won't do what I need. It adds a lot
of additional DOM elements and it's still not a put-on-and-forget-about-it
solution. You would have to pregenerate geometric shapes as developer. Not
necessarily a fun thing to do for most web devs.

Cheers,
Paul

Am 22.06.11 14:16 schrieb "Charles McCathieNevile" unter
<chaals@opera.com>:

>On Wed, 22 Jun 2011 13:06:08 +0200, Paul Bakaus <pbakaus@zynga.com> wrote:
>
>> Hi everyone,
>>
>> We at Zynga use hit testing to a great extend in our isometric games and
>> have long searched for a solution that removes some of the processing
>> burden from us. I therefore very much disagree that hit testing should
>> be up to the js developer. If possible, this is a perfect candidate for
>> a job the browser can help us with. And by "us", I am probably talking
>> about almost every JS game developer.
>
>What about placing a transparent image over the top of your canvas, and
>using an image map in it. Things that need to be clickable can be updated
>according to whatever you're drawing, things that don't remain purely
>immediate mode.
>
>I haven't tried it, but I've been thinking about it (and about how the
>whole discussion reminds me of ISSUE-105 [1]) and I ran across something
>that demonstrates the technique [2] (although I think the purpose of that
>demo was something else, and whatever it was trying to do seems horribly
>complex to me).
>
>[1] http://www.w3.org/html/wg/tracker/issues/105
>[2] http://zreference.com/image-map-canvas/
>
>cheers
>
>Chaals
>
>> I proposed an extension to CSS in April that a lot of folks I talked to
>> liked, as it is both simple and effective for simple hit testing on DOM
>> elements. You can read the summary in this thread:
>> http://lists.w3.org/Archives/Public/www-style/2011Apr/0558.html
>>
>> Paul
>>
>> Von: Charles Pritchard <chuck@jumis.com<mailto:chuck@jumis.com>>
>> Datum: Tue, 21 Jun 2011 21:36:57 -0700
>> An: Richard Schwerdtfeger <schwer@us.ibm.com<mailto:schwer@us.ibm.com>>
>> Cc: Frank Olivier
>> <Frank.Olivier@microsoft.com<mailto:Frank.Olivier@microsoft.com>>,
>> Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>,
>> "david.bolter@gmail.com<mailto:david.bolter@gmail.com>"
>> <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>,
>> "Mike@w3.org<mailto:Mike@w3.org>" <Mike@w3.org<mailto:Mike@w3.org>>,
>> "public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>"
>> <public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>>,
>> "public-html@w3.org<mailto:public-html@w3.org>"
>> <public-html@w3.org<mailto:public-html@w3.org>>,
>> "public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>"
>> <public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>>,
>> "public-html-request@w3.org<mailto:public-html-request@w3.org>"
>> <public-html-request@w3.org<mailto:public-html-request@w3.org>>
>> Betreff: Re: hit testing and retained graphics
>>
>> This is a lengthy e-mail. Summary; three parts:
>>
>> 1. Retained Path objects are ubiquitous in basic 2d drawing APIs;
>> they are efficient, they are managed by the developer, and
>> they are necessary for accessibility.
>>
>> 2. Basic additions to the canvas spec can introduce path objects
>> which can be bound to DOM elements.
>>
>> 3. An example of three buttons in a canvas element is shown,
>> demonstrating the API.
>>
>> 4. Some usage notes.
>>
>> ....................
>>
>> Part 1. Path objects are common in drawing APIs.
>>
>> I've worked a little with Richard's proposal, and done a quick review on
>> the major drawing APIs out there that
>> work on the same level of abstraction as Canvas.
>>
>>
>>http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/
>>display/GraphicsPath.html
>>
>>http://msdn.microsoft.com/en-us/library/system.drawing.drawing2d.pathdata
>>_members(v=vs.85).aspx
>>
>>http://download.oracle.com/javase/7/docs/api/java/awt/geom/PathIterator.h
>>tml
>>
>>http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Co
>>coaDrawingGuide/Paths/Paths.html
>> "If you draw the same content repeatedly ... [it] is usually more
>> efficient to retain an existing NSBezierPath object than to recreate it
>> during each drawing cycle."
>>
>> That last excerpt applies to a few rendering APIs, not just Apple's
>> Cocoa.
>> There's precedent in each of these APIs for a Path data object.
>> WebGL uses path data by typed arrays. It does not include curves.
>> SVG has normalized path types:
>> http://msdn.microsoft.com/en-us/library/ff971968(v=vs.85).aspx
>>
>> I do not think we need "retained graphics" in canvas -- but it'd be a
>> huge boost to developers,
>> and to accessibility, if we can have a means for sharing path data
>> between layers.
>> The HTML map tag is insufficient... We're looking for a replacement.
>>
>> Richard has put forward a few proposals, as have I... If we can get
>> some traction with the group, I'm sure we can add a great feature
>> to Canvas, making life easier on developers and AT vendors alike.
>>
>> Developers have told me that they want mouse events delegated
>> to the shadow dom, based on paths they've bound. Richard and I
>> are most concerned with getting path data to ATs; though Richard
>> has a warmth in his heart for making things easier on developers.
>>
>> Many vendors are concerned about mixing retained-mode concepts
>> with Canvas's immediate mode API.
>> I hope that by focusing only on static path data, we can move past
>> that concern. I also hope that by introducing new methods, we can
>> see improved efficiency and accessibility in Canvas applications.
>>
>> Please take the following in good faith, and send the list feedback,
>> so we can take that feedback, apply it, and push the proposal into
>> the view of a wider audience.
>>
>> Part 2. Creating a CanvasPath object and stringify methods in the spec.
>>
>> I've done a basic supplemental. setElementFromPath binds the current
>> path onto an element, with an optional zIndex. I've also done a
>> stringify,
>> supplemental, which would make life a lot easier on developers when it
>> comes to getting a string for the current path or transformation matrix.
>>
>> // CanvasPath extension
>> [Supplemental]
>> interface CanvasRenderingContext2D {
>>     void beginFromPath(in CanvasPath path) ;
>>     void setElementPath(in Element element, in optional DOMString
>> zIndex) ;
>>     CanvasPath getCurrentPath();
>> }
>>
>> interface CanvasPath {
>>     // opaque object
>> }
>>
>> // Stringify extensions, defined by SVG normalized paths and CSSMatrix.
>> [Supplemental]
>> interface CanvasRenderingContext2D {
>>     void beginFromPath(in normalizedPathSegList path) ;
>>     void beginFromPath(in DOMString path) ;
>>     CSSMatrix getCurrentMatrix();
>>     void setCurrentMatrix(in CSSMatrix matrix);
>>     void setCurrentMatrix(in DOMString matrix);
>> }
>> [Supplemental]
>> interface CanvasPath {
>>     stringifier; // returns normalized path
>> }
>>
>>
>> Part 3. Example of an accessible Canvas, keyboard and pointer device.
>> /*
>>  Draw three buttons, 100x100 pixels each, spaced 40px apart
>>     If a button is in focus, set its zIndex higher than the other
>> buttons,
>>     in case of overlap if spacing is changed.
>> */
>>
>> var canvas = document.getElementById(ŒcanvasTest¹); canvas.width += 0;
>> // reset state.
>> var ctx = canvas.getContext(¹2d¹);
>> var buttons = { 'a': 'Button A', 'b': 'Button B','c': 'Button C' }
>> var width = 100; var height=30; var padding = 40;
>> ctx.font = '20px sans-serif';
>> ctx.beginPath();
>> ctx.rect(0,0,width,height);
>> var path = ctx.getCurrentPath();
>> for(var i in buttons) {
>>     var button = document.getElementById('button_'+i);
>>     if(!button) {
>>         button = document.createElement('input');
>>         button.setAttribute('id', 'button_'+i);
>>         button.setAttribute('name', i);
>>         button.setAttribute('value', buttons[i]);
>>         button.setAttribute('type','submit');
>>         if(0) button.onclick = function() { alert("It works!"); };
>>         if(0) button.onfocus = function() {
>>             if(button.getAttribute('data-myeasypathdata')) {
>>                 button.parentNode.getContext('2d').beginFromPath(
>> button.getAttribute('data-myeasypathdata') );
>>
>>button.parentNode.getContext('2d').drawFocusRing(button);
>>             }
>>         };
>>         canvas.appendChild(button);
>>     }
>>     var isFocused = document.activeElement == button;
>>     var zIndex = isFocused ? "1" : "0";
>>     ctx.beginFromPath(path);
>>     ctx.setElementPath(button, zIndex);
>>     if(isFocused) ctx.drawFocusRing(button);
>>     if(0) button.setAttribute('data-myeasypathdata',
>> ctx.getCurrentPath());
>>     ctx.fillStyle = 'black';
>>     ctx.fill();
>>     ctx.fillStyle = 'white';
>>     var textBaseline = ....;
>>     ctx.fillText(buttons[i], ( width - ctx.measureText(buttons[i]).width
>> ) / 2, textBaseline);
>>     ctx.translate(width + padding, 0);
>> }
>>
>>
>> Part 4. Usage notes.
>>
>> Stringify options allow developers more room for loose coupling of
>> events;
>> they also allow better interop between SVG and Canvas... They help with
>> debugging;
>> they can reduce the overhead of complex objects, by reducing the number
>> of method
>> calls, considerably.
>>
>> z-index is necessary for coordinate/pointer-based interfaces; things
>> overlap.
>>
>> Transformations will be used heavily in animations. Transformations are
>> live, in canvas.
>> Notice that only one translate method is called, incrementally moving
>> the origin along the x axis.
>>
>>
>> -----
>>
>>
>> There are several ATs which require coordinates / paths ahead of time.
>> Some ATs get by this issue by focusing on an element before requesting
>> the bounding box.
>>
>> ViewPlus produces the "IVEO Hands-on-Learning System":
>> http://www.viewplus.com/
>>
>> Apple produces Mobile Safari, with "VoiceOver":
>> http://www.apple.com/accessibility/iphone/vision.html
>>
>> Both of these products weighed heavily in the design of the proposal.
>>
>> Input is appreciated.
>>
>> -Charles
>>
>>
>> On 6/20/2011 3:45 PM, Richard Schwerdtfeger wrote:
>>
>> I have some concerns about this. It means:
>>
>> - The author has to do the transformations
>> - We have to add DOM attributes
>> - The mouse events are routed to canvas and not the sub-DOM elements
>> where the keyboard handling is going on. The user agent could process
>> the mouse events at canvas and then propagate them to the corresponding
>> DOM object in the subtree.
>>
>> Your proposal does have the advantage that the bounds of the objects are
>> in the DOM but for HTML we don't have this for any of the DOM elements
>> now.
>>
>> What I was thinking was the following:
>>
>> - Today Canvas has the notion of a context
>> - We allow the author to have the same context (with methods) for each
>> drawing object and apply a bounds and z-index as you suggest.
>> - We then bind each drawing object to the canvas subtree DOM element:
>>
>> So each drawing object would be an instance of a canvas context with
>> methods were we do something like:
>>
>> 1. we assume that the canvas element when the page is created is an
>> instance of a canvasObject (having a context)
>> 2. we assume that drawingOjects are a subclass of canvasObject that
>> support all the canvas2DAPI in canvasObject with some additions such as:
>> - ZIndex attribute
>> - a bounding drawing path and methods for modifying them
>> - a method for associating the drawingObject with a canvas subtree DOM
>> element.
>> 3. we add an method to canvas that says addDrawObject.
>> On the canvas element we have the following:
>>
>> var canvas = document.getElementById(ŒcanvasTest¹);
>> if (canvas.getContext){
>> var ctx = canvas.getContext(¹2d¹);
>> DO = new drawingObject();
>> dctx= DO.getContext('2d');
>> dctx.ZIndex="2";
>> dctx.beginPath();
>> dctx.moveTo(130,100);
>> dctx.lineTo...
>> ...
>> dctx.setPathtoBounds();
>> dctx.setDOMSubtreeNode(foo); //Foo is a subtree node where keyboard
>> events go to and we do our accessibility enablement to populate the
>> accessibility tree
>> //Internally the user agents maps the bounding rectangle for the
>> accessible of the DOM object as a best fit rectangle of the
>> //path used to form the bounds of the drawing object
>> ctx.addDrawingObject(DO);
>> }
>>
>> Now the user agent the information needed to do the hit testing and
>> retained mode graphics (I am oversimplifying for illustration purposes)
>> to be able to track the pointing device input and routing it to the same
>> DOM objects that process the keyboard and all the other accessibility
>> information. This includes hit testing.
>>
>> Mike provided feedback on the HTML A11Y call that authors did not want
>> to do the hit testing themselves.
>> Rich Schwerdtfeger
>> CTO Accessibility Software Group
>>
>> [cid:part1.01000009.05010600@jumis.com]Frank Olivier ---06/20/2011
>> 11:01:54 AM---I would leave hit testing up to the (javascript) author. I
>> would recommend that they set existing x,
>>
>> From: Frank Olivier
>> <Frank.Olivier@microsoft.com><mailto:Frank.Olivier@microsoft.com>
>> To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
>> "chuck@jumis.com"<mailto:chuck@jumis.com>
>> <chuck@jumis.com><mailto:chuck@jumis.com>,
>> "Mike@w3.org"<mailto:Mike@w3.org> <Mike@w3.org><mailto:Mike@w3.org>,
>> "david.bolter@gmail.com"<mailto:david.bolter@gmail.com>
>> <david.bolter@gmail.com><mailto:david.bolter@gmail.com>, Cynthia Shelly
>> <cyns@microsoft.com><mailto:cyns@microsoft.com>
>> Cc: "public-canvas-api@w3.org"<mailto:public-canvas-api@w3.org>
>> <public-canvas-api@w3.org><mailto:public-canvas-api@w3.org>,
>> "public-html-a11y@w3.org"<mailto:public-html-a11y@w3.org>
>> <public-html-a11y@w3.org><mailto:public-html-a11y@w3.org>,
>> "public-html@w3.org"<mailto:public-html@w3.org>
>> <public-html@w3.org><mailto:public-html@w3.org>
>> Date: 06/20/2011 11:01 AM
>> Subject: RE: hit testing and retained graphics
>> Sent by: public-html-request@w3.org<mailto:public-html-request@w3.org>
>>
>> ________________________________
>>
>>
>>
>> I would leave hit testing up to the (javascript) author. I would
>> recommend that they set existing x,y position, z-index attributes on the
>> DOM objects in the canvas subtree to report what the UI 'looks like' to
>> AT tools. This way, the AT tools don't need to change - this part of the
>> DOM is no different to them than any other part - and authors need to be
>> annotating canvas DOM objects with correct information anyway (labels,
>> aria attributes, etc).
>>
>> From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
>> Sent: Friday, June 17, 2011 11:42 AM
>> To: chuck@jumis.com<mailto:chuck@jumis.com>; Frank Olivier;
>> Mike@w3.org<mailto:Mike@w3.org>;
>> david.bolter@gmail.com<mailto:david.bolter@gmail.com>; Cynthia Shelly
>> Cc: public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>;
>> public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>;
>> public-html@w3.org<mailto:public-html@w3.org>
>> Subject: hit testing and retained graphics
>>
>> Charles, Frank, Mike,
>>
>> I am back from vacation. How far do we need to go with hit testing?
>> Right now I am looking at associating a closed draw path with a DOM
>> object in the canvas subtree. We would then need to address the routing
>> of pointing device input events to the DOM object. The drawing path can
>> be used to provide bound information to platform accessibility API.
>>
>> Do we need to bind any other drawing properties to the canvas object -
>> similar to the way device context's are handled on graphic subsystems
>> like Windows?
>>
>> Mike, I am including you as before I went on vacation you indicated that
>> a number of developers desired this feature and wanted to be involved.
>>
>> Rich
>>
>>
>> Rich Schwerdtfeger
>> CTO Accessibility Software Group
>>
>>
>>
>
>
>--
>Charles McCathieNevile  Opera Software, Standards Group
>     je parle français -- hablo español -- jeg lærer norsk
>http://my.opera.com/chaals       Try Opera: http://www.opera.com
>

Received on Wednesday, 22 June 2011 12:26:19 UTC