Re: hit testing and retained graphics



Am 22.06.11 19:53 schrieb "Charles Pritchard" unter <chuck@jumis.com>:

>Paul,
>
>The proposal is intended for vector graphics managed by canvas. I
>understand that your development team simply uses PNG files an associates
>them for hit testing; you've proposed CSS attributes be applied so that
>the transparent sections of such images pass thru pointer events.
>
>It's difficult to make those objects 'accessible' beyond simply sharing
>the bounding box of the png with the Accessibility API.

I agree. I only brought it up as technically, we're trying to solve
something quite similar and might benefit from each others thoughts.

>
>
>-Charles
>
>
>On Jun 22, 2011, at 5:25 AM, Paul Bakaus <pbakaus@zynga.com> wrote:
>
>> Hi Charles,
>>
>> I've been thinking about this but it won't do what I need. It adds a lot
>> of additional DOM elements and it's still not a
>>put-on-and-forget-about-it
>> solution. You would have to pregenerate geometric shapes as developer.
>>Not
>> necessarily a fun thing to do for most web devs.
>>
>> Cheers,
>> Paul
>>
>> Am 22.06.11 14:16 schrieb "Charles McCathieNevile" unter
>> <chaals@opera.com>:
>>
>>> On Wed, 22 Jun 2011 13:06:08 +0200, Paul Bakaus <pbakaus@zynga.com>
>>>wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> We at Zynga use hit testing to a great extend in our isometric games
>>>>and
>>>> have long searched for a solution that removes some of the processing
>>>> burden from us. I therefore very much disagree that hit testing should
>>>> be up to the js developer. If possible, this is a perfect candidate
>>>>for
>>>> a job the browser can help us with. And by "us", I am probably talking
>>>> about almost every JS game developer.
>>>
>>> What about placing a transparent image over the top of your canvas, and
>>> using an image map in it. Things that need to be clickable can be
>>>updated
>>> according to whatever you're drawing, things that don't remain purely
>>> immediate mode.
>>>
>>> I haven't tried it, but I've been thinking about it (and about how the
>>> whole discussion reminds me of ISSUE-105 [1]) and I ran across
>>>something
>>> that demonstrates the technique [2] (although I think the purpose of
>>>that
>>> demo was something else, and whatever it was trying to do seems
>>>horribly
>>> complex to me).
>>>
>>> [1] http://www.w3.org/html/wg/tracker/issues/105

>>> [2] http://zreference.com/image-map-canvas/

>>>
>>> cheers
>>>
>>> Chaals
>>>
>>>> I proposed an extension to CSS in April that a lot of folks I talked
>>>>to
>>>> liked, as it is both simple and effective for simple hit testing on
>>>>DOM
>>>> elements. You can read the summary in this thread:
>>>> http://lists.w3.org/Archives/Public/www-style/2011Apr/0558.html

>>>>
>>>> Paul
>>>>
>>>> Von: Charles Pritchard <chuck@jumis.com<mailto:chuck@jumis.com>>
>>>> Datum: Tue, 21 Jun 2011 21:36:57 -0700
>>>> An: Richard Schwerdtfeger
>>>><schwer@us.ibm.com<mailto:schwer@us.ibm.com>>
>>>> Cc: Frank Olivier
>>>> <Frank.Olivier@microsoft.com<mailto:Frank.Olivier@microsoft.com>>,
>>>> Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>,
>>>> "david.bolter@gmail.com<mailto:david.bolter@gmail.com>"
>>>> <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>,
>>>> "Mike@w3.org<mailto:Mike@w3.org>" <Mike@w3.org<mailto:Mike@w3.org>>,
>>>> "public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>"
>>>> <public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>>,
>>>> "public-html@w3.org<mailto:public-html@w3.org>"
>>>> <public-html@w3.org<mailto:public-html@w3.org>>,
>>>> "public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>"
>>>> <public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>>,
>>>> "public-html-request@w3.org<mailto:public-html-request@w3.org>"
>>>> <public-html-request@w3.org<mailto:public-html-request@w3.org>>
>>>> Betreff: Re: hit testing and retained graphics
>>>>
>>>> This is a lengthy e-mail. Summary; three parts:
>>>>
>>>> 1. Retained Path objects are ubiquitous in basic 2d drawing APIs;
>>>> they are efficient, they are managed by the developer, and
>>>> they are necessary for accessibility.
>>>>
>>>> 2. Basic additions to the canvas spec can introduce path objects
>>>> which can be bound to DOM elements.
>>>>
>>>> 3. An example of three buttons in a canvas element is shown,
>>>> demonstrating the API.
>>>>
>>>> 4. Some usage notes.
>>>>
>>>> ....................
>>>>
>>>> Part 1. Path objects are common in drawing APIs.
>>>>
>>>> I've worked a little with Richard's proposal, and done a quick review
>>>>on
>>>> the major drawing APIs out there that
>>>> work on the same level of abstraction as Canvas.
>>>>
>>>>
>>>>
>>>>http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flas

>>>>h/
>>>> display/GraphicsPath.html
>>>>
>>>>
>>>>http://msdn.microsoft.com/en-us/library/system.drawing.drawing2d.pathda

>>>>ta
>>>> _members(v=vs.85).aspx
>>>>
>>>>
>>>>http://download.oracle.com/javase/7/docs/api/java/awt/geom/PathIterator

>>>>.h
>>>> tml
>>>>
>>>>
>>>>http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/

>>>>Co
>>>> coaDrawingGuide/Paths/Paths.html
>>>> "If you draw the same content repeatedly ... [it] is usually more
>>>> efficient to retain an existing NSBezierPath object than to recreate
>>>>it
>>>> during each drawing cycle."
>>>>
>>>> That last excerpt applies to a few rendering APIs, not just Apple's
>>>> Cocoa.
>>>> There's precedent in each of these APIs for a Path data object.
>>>> WebGL uses path data by typed arrays. It does not include curves.
>>>> SVG has normalized path types:
>>>> http://msdn.microsoft.com/en-us/library/ff971968(v=vs.85).aspx
>>>>
>>>> I do not think we need "retained graphics" in canvas -- but it'd be a
>>>> huge boost to developers,
>>>> and to accessibility, if we can have a means for sharing path data
>>>> between layers.
>>>> The HTML map tag is insufficient... We're looking for a replacement.
>>>>
>>>> Richard has put forward a few proposals, as have I... If we can get
>>>> some traction with the group, I'm sure we can add a great feature
>>>> to Canvas, making life easier on developers and AT vendors alike.
>>>>
>>>> Developers have told me that they want mouse events delegated
>>>> to the shadow dom, based on paths they've bound. Richard and I
>>>> are most concerned with getting path data to ATs; though Richard
>>>> has a warmth in his heart for making things easier on developers.
>>>>
>>>> Many vendors are concerned about mixing retained-mode concepts
>>>> with Canvas's immediate mode API.
>>>> I hope that by focusing only on static path data, we can move past
>>>> that concern. I also hope that by introducing new methods, we can
>>>> see improved efficiency and accessibility in Canvas applications.
>>>>
>>>> Please take the following in good faith, and send the list feedback,
>>>> so we can take that feedback, apply it, and push the proposal into
>>>> the view of a wider audience.
>>>>
>>>> Part 2. Creating a CanvasPath object and stringify methods in the
>>>>spec.
>>>>
>>>> I've done a basic supplemental. setElementFromPath binds the current
>>>> path onto an element, with an optional zIndex. I've also done a
>>>> stringify,
>>>> supplemental, which would make life a lot easier on developers when it
>>>> comes to getting a string for the current path or transformation
>>>>matrix.
>>>>
>>>> // CanvasPath extension
>>>> [Supplemental]
>>>> interface CanvasRenderingContext2D {
>>>>    void beginFromPath(in CanvasPath path) ;
>>>>    void setElementPath(in Element element, in optional DOMString
>>>> zIndex) ;
>>>>    CanvasPath getCurrentPath();
>>>> }
>>>>
>>>> interface CanvasPath {
>>>>    // opaque object
>>>> }
>>>>
>>>> // Stringify extensions, defined by SVG normalized paths and
>>>>CSSMatrix.
>>>> [Supplemental]
>>>> interface CanvasRenderingContext2D {
>>>>    void beginFromPath(in normalizedPathSegList path) ;
>>>>    void beginFromPath(in DOMString path) ;
>>>>    CSSMatrix getCurrentMatrix();
>>>>    void setCurrentMatrix(in CSSMatrix matrix);
>>>>    void setCurrentMatrix(in DOMString matrix);
>>>> }
>>>> [Supplemental]
>>>> interface CanvasPath {
>>>>    stringifier; // returns normalized path
>>>> }
>>>>
>>>>
>>>> Part 3. Example of an accessible Canvas, keyboard and pointer device.
>>>> /*
>>>> Draw three buttons, 100x100 pixels each, spaced 40px apart
>>>>    If a button is in focus, set its zIndex higher than the other
>>>> buttons,
>>>>    in case of overlap if spacing is changed.
>>>> */
>>>>
>>>> var canvas = document.getElementById(ŒcanvasTest¹); canvas.width += 0;
>>>> // reset state.
>>>> var ctx = canvas.getContext(¹2d¹);
>>>> var buttons = { 'a': 'Button A', 'b': 'Button B','c': 'Button C' }
>>>> var width = 100; var height=30; var padding = 40;
>>>> ctx.font = '20px sans-serif';
>>>> ctx.beginPath();
>>>> ctx.rect(0,0,width,height);
>>>> var path = ctx.getCurrentPath();
>>>> for(var i in buttons) {
>>>>    var button = document.getElementById('button_'+i);
>>>>    if(!button) {
>>>>        button = document.createElement('input');
>>>>        button.setAttribute('id', 'button_'+i);
>>>>        button.setAttribute('name', i);
>>>>        button.setAttribute('value', buttons[i]);
>>>>        button.setAttribute('type','submit');
>>>>        if(0) button.onclick = function() { alert("It works!"); };
>>>>        if(0) button.onfocus = function() {
>>>>            if(button.getAttribute('data-myeasypathdata')) {
>>>>                button.parentNode.getContext('2d').beginFromPath(
>>>> button.getAttribute('data-myeasypathdata') );
>>>>
>>>> button.parentNode.getContext('2d').drawFocusRing(button);
>>>>            }
>>>>        };
>>>>        canvas.appendChild(button);
>>>>    }
>>>>    var isFocused = document.activeElement == button;
>>>>    var zIndex = isFocused ? "1" : "0";
>>>>    ctx.beginFromPath(path);
>>>>    ctx.setElementPath(button, zIndex);
>>>>    if(isFocused) ctx.drawFocusRing(button);
>>>>    if(0) button.setAttribute('data-myeasypathdata',
>>>> ctx.getCurrentPath());
>>>>    ctx.fillStyle = 'black';
>>>>    ctx.fill();
>>>>    ctx.fillStyle = 'white';
>>>>    var textBaseline = ....;
>>>>    ctx.fillText(buttons[i], ( width -
>>>>ctx.measureText(buttons[i]).width
>>>> ) / 2, textBaseline);
>>>>    ctx.translate(width + padding, 0);
>>>> }
>>>>
>>>>
>>>> Part 4. Usage notes.
>>>>
>>>> Stringify options allow developers more room for loose coupling of
>>>> events;
>>>> they also allow better interop between SVG and Canvas... They help
>>>>with
>>>> debugging;
>>>> they can reduce the overhead of complex objects, by reducing the
>>>>number
>>>> of method
>>>> calls, considerably.
>>>>
>>>> z-index is necessary for coordinate/pointer-based interfaces; things
>>>> overlap.
>>>>
>>>> Transformations will be used heavily in animations. Transformations
>>>>are
>>>> live, in canvas.
>>>> Notice that only one translate method is called, incrementally moving
>>>> the origin along the x axis.
>>>>
>>>>
>>>> -----
>>>>
>>>>
>>>> There are several ATs which require coordinates / paths ahead of time.
>>>> Some ATs get by this issue by focusing on an element before requesting
>>>> the bounding box.
>>>>
>>>> ViewPlus produces the "IVEO Hands-on-Learning System":
>>>> http://www.viewplus.com/

>>>>
>>>> Apple produces Mobile Safari, with "VoiceOver":
>>>> http://www.apple.com/accessibility/iphone/vision.html

>>>>
>>>> Both of these products weighed heavily in the design of the proposal.
>>>>
>>>> Input is appreciated.
>>>>
>>>> -Charles
>>>>
>>>>
>>>> On 6/20/2011 3:45 PM, Richard Schwerdtfeger wrote:
>>>>
>>>> I have some concerns about this. It means:
>>>>
>>>> - The author has to do the transformations
>>>> - We have to add DOM attributes
>>>> - The mouse events are routed to canvas and not the sub-DOM elements
>>>> where the keyboard handling is going on. The user agent could process
>>>> the mouse events at canvas and then propagate them to the
>>>>corresponding
>>>> DOM object in the subtree.
>>>>
>>>> Your proposal does have the advantage that the bounds of the objects
>>>>are
>>>> in the DOM but for HTML we don't have this for any of the DOM elements
>>>> now.
>>>>
>>>> What I was thinking was the following:
>>>>
>>>> - Today Canvas has the notion of a context
>>>> - We allow the author to have the same context (with methods) for each
>>>> drawing object and apply a bounds and z-index as you suggest.
>>>> - We then bind each drawing object to the canvas subtree DOM element:
>>>>
>>>> So each drawing object would be an instance of a canvas context with
>>>> methods were we do something like:
>>>>
>>>> 1. we assume that the canvas element when the page is created is an
>>>> instance of a canvasObject (having a context)
>>>> 2. we assume that drawingOjects are a subclass of canvasObject that
>>>> support all the canvas2DAPI in canvasObject with some additions such
>>>>as:
>>>> - ZIndex attribute
>>>> - a bounding drawing path and methods for modifying them
>>>> - a method for associating the drawingObject with a canvas subtree DOM
>>>> element.
>>>> 3. we add an method to canvas that says addDrawObject.
>>>> On the canvas element we have the following:
>>>>
>>>> var canvas = document.getElementById(ŒcanvasTest¹);
>>>> if (canvas.getContext){
>>>> var ctx = canvas.getContext(¹2d¹);
>>>> DO = new drawingObject();
>>>> dctx= DO.getContext('2d');
>>>> dctx.ZIndex="2";
>>>> dctx.beginPath();
>>>> dctx.moveTo(130,100);
>>>> dctx.lineTo...
>>>> ...
>>>> dctx.setPathtoBounds();
>>>> dctx.setDOMSubtreeNode(foo); //Foo is a subtree node where keyboard
>>>> events go to and we do our accessibility enablement to populate the
>>>> accessibility tree
>>>> //Internally the user agents maps the bounding rectangle for the
>>>> accessible of the DOM object as a best fit rectangle of the
>>>> //path used to form the bounds of the drawing object
>>>> ctx.addDrawingObject(DO);
>>>> }
>>>>
>>>> Now the user agent the information needed to do the hit testing and
>>>> retained mode graphics (I am oversimplifying for illustration
>>>>purposes)
>>>> to be able to track the pointing device input and routing it to the
>>>>same
>>>> DOM objects that process the keyboard and all the other accessibility
>>>> information. This includes hit testing.
>>>>
>>>> Mike provided feedback on the HTML A11Y call that authors did not want
>>>> to do the hit testing themselves.
>>>> Rich Schwerdtfeger
>>>> CTO Accessibility Software Group
>>>>
>>>> [cid:part1.01000009.05010600@jumis.com]Frank Olivier ---06/20/2011
>>>> 11:01:54 AM---I would leave hit testing up to the (javascript)
>>>>author. I
>>>> would recommend that they set existing x,
>>>>
>>>> From: Frank Olivier
>>>> <Frank.Olivier@microsoft.com><mailto:Frank.Olivier@microsoft.com>
>>>> To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
>>>> "chuck@jumis.com"<mailto:chuck@jumis.com>
>>>> <chuck@jumis.com><mailto:chuck@jumis.com>,
>>>> "Mike@w3.org"<mailto:Mike@w3.org> <Mike@w3.org><mailto:Mike@w3.org>,
>>>> "david.bolter@gmail.com"<mailto:david.bolter@gmail.com>
>>>> <david.bolter@gmail.com><mailto:david.bolter@gmail.com>, Cynthia
>>>>Shelly
>>>> <cyns@microsoft.com><mailto:cyns@microsoft.com>
>>>> Cc: "public-canvas-api@w3.org"<mailto:public-canvas-api@w3.org>
>>>> <public-canvas-api@w3.org><mailto:public-canvas-api@w3.org>,
>>>> "public-html-a11y@w3.org"<mailto:public-html-a11y@w3.org>
>>>> <public-html-a11y@w3.org><mailto:public-html-a11y@w3.org>,
>>>> "public-html@w3.org"<mailto:public-html@w3.org>
>>>> <public-html@w3.org><mailto:public-html@w3.org>
>>>> Date: 06/20/2011 11:01 AM
>>>> Subject: RE: hit testing and retained graphics
>>>> Sent by: public-html-request@w3.org<mailto:public-html-request@w3.org>
>>>>
>>>> ________________________________
>>>>
>>>>
>>>>
>>>> I would leave hit testing up to the (javascript) author. I would
>>>> recommend that they set existing x,y position, z-index attributes on
>>>>the
>>>> DOM objects in the canvas subtree to report what the UI 'looks like'
>>>>to
>>>> AT tools. This way, the AT tools don't need to change - this part of
>>>>the
>>>> DOM is no different to them than any other part - and authors need to
>>>>be
>>>> annotating canvas DOM objects with correct information anyway (labels,
>>>> aria attributes, etc).
>>>>
>>>> From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
>>>> Sent: Friday, June 17, 2011 11:42 AM
>>>> To: chuck@jumis.com<mailto:chuck@jumis.com>; Frank Olivier;
>>>> Mike@w3.org<mailto:Mike@w3.org>;
>>>> david.bolter@gmail.com<mailto:david.bolter@gmail.com>; Cynthia Shelly
>>>> Cc: public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>;
>>>> public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>;
>>>> public-html@w3.org<mailto:public-html@w3.org>
>>>> Subject: hit testing and retained graphics
>>>>
>>>> Charles, Frank, Mike,
>>>>
>>>> I am back from vacation. How far do we need to go with hit testing?
>>>> Right now I am looking at associating a closed draw path with a DOM
>>>> object in the canvas subtree. We would then need to address the
>>>>routing
>>>> of pointing device input events to the DOM object. The drawing path
>>>>can
>>>> be used to provide bound information to platform accessibility API.
>>>>
>>>> Do we need to bind any other drawing properties to the canvas object -
>>>> similar to the way device context's are handled on graphic subsystems
>>>> like Windows?
>>>>
>>>> Mike, I am including you as before I went on vacation you indicated
>>>>that
>>>> a number of developers desired this feature and wanted to be involved.
>>>>
>>>> Rich
>>>>
>>>>
>>>> Rich Schwerdtfeger
>>>> CTO Accessibility Software Group
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Charles McCathieNevile  Opera Software, Standards Group
>>>    je parle français -- hablo español -- jeg lærer norsk
>>> http://my.opera.com/chaals       Try Opera: http://www.opera.com

>>>
>>
>>

Received on Thursday, 23 June 2011 07:38:28 UTC