- From: Paul Bakaus <pbakaus@zynga.com>
- Date: Thu, 23 Jun 2011 00:37:54 -0700
- To: Charles Pritchard <chuck@jumis.com>
- CC: Charles McCathieNevile <chaals@opera.com>, Richard Schwerdtfeger <schwer@us.ibm.com>, Frank Olivier <Frank.Olivier@microsoft.com>, Cynthia Shelly <cyns@microsoft.com>, "david.bolter@gmail.com" <david.bolter@gmail.com>, "Mike@w3.org" <Mike@w3.org>, "public-canvas-api@w3.org" <public-canvas-api@w3.org>, "public-html@w3.org" <public-html@w3.org>, "public-html-a11y@w3.org" <public-html-a11y@w3.org>
Am 22.06.11 19:53 schrieb "Charles Pritchard" unter <chuck@jumis.com>: >Paul, > >The proposal is intended for vector graphics managed by canvas. I >understand that your development team simply uses PNG files an associates >them for hit testing; you've proposed CSS attributes be applied so that >the transparent sections of such images pass thru pointer events. > >It's difficult to make those objects 'accessible' beyond simply sharing >the bounding box of the png with the Accessibility API. I agree. I only brought it up as technically, we're trying to solve something quite similar and might benefit from each others thoughts. > > >-Charles > > >On Jun 22, 2011, at 5:25 AM, Paul Bakaus <pbakaus@zynga.com> wrote: > >> Hi Charles, >> >> I've been thinking about this but it won't do what I need. It adds a lot >> of additional DOM elements and it's still not a >>put-on-and-forget-about-it >> solution. You would have to pregenerate geometric shapes as developer. >>Not >> necessarily a fun thing to do for most web devs. >> >> Cheers, >> Paul >> >> Am 22.06.11 14:16 schrieb "Charles McCathieNevile" unter >> <chaals@opera.com>: >> >>> On Wed, 22 Jun 2011 13:06:08 +0200, Paul Bakaus <pbakaus@zynga.com> >>>wrote: >>> >>>> Hi everyone, >>>> >>>> We at Zynga use hit testing to a great extend in our isometric games >>>>and >>>> have long searched for a solution that removes some of the processing >>>> burden from us. I therefore very much disagree that hit testing should >>>> be up to the js developer. If possible, this is a perfect candidate >>>>for >>>> a job the browser can help us with. And by "us", I am probably talking >>>> about almost every JS game developer. >>> >>> What about placing a transparent image over the top of your canvas, and >>> using an image map in it. Things that need to be clickable can be >>>updated >>> according to whatever you're drawing, things that don't remain purely >>> immediate mode. >>> >>> I haven't tried it, but I've been thinking about it (and about how the >>> whole discussion reminds me of ISSUE-105 [1]) and I ran across >>>something >>> that demonstrates the technique [2] (although I think the purpose of >>>that >>> demo was something else, and whatever it was trying to do seems >>>horribly >>> complex to me). >>> >>> [1] http://www.w3.org/html/wg/tracker/issues/105 >>> [2] http://zreference.com/image-map-canvas/ >>> >>> cheers >>> >>> Chaals >>> >>>> I proposed an extension to CSS in April that a lot of folks I talked >>>>to >>>> liked, as it is both simple and effective for simple hit testing on >>>>DOM >>>> elements. You can read the summary in this thread: >>>> http://lists.w3.org/Archives/Public/www-style/2011Apr/0558.html >>>> >>>> Paul >>>> >>>> Von: Charles Pritchard <chuck@jumis.com<mailto:chuck@jumis.com>> >>>> Datum: Tue, 21 Jun 2011 21:36:57 -0700 >>>> An: Richard Schwerdtfeger >>>><schwer@us.ibm.com<mailto:schwer@us.ibm.com>> >>>> Cc: Frank Olivier >>>> <Frank.Olivier@microsoft.com<mailto:Frank.Olivier@microsoft.com>>, >>>> Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>, >>>> "david.bolter@gmail.com<mailto:david.bolter@gmail.com>" >>>> <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>, >>>> "Mike@w3.org<mailto:Mike@w3.org>" <Mike@w3.org<mailto:Mike@w3.org>>, >>>> "public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>" >>>> <public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>>, >>>> "public-html@w3.org<mailto:public-html@w3.org>" >>>> <public-html@w3.org<mailto:public-html@w3.org>>, >>>> "public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>" >>>> <public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>>, >>>> "public-html-request@w3.org<mailto:public-html-request@w3.org>" >>>> <public-html-request@w3.org<mailto:public-html-request@w3.org>> >>>> Betreff: Re: hit testing and retained graphics >>>> >>>> This is a lengthy e-mail. Summary; three parts: >>>> >>>> 1. Retained Path objects are ubiquitous in basic 2d drawing APIs; >>>> they are efficient, they are managed by the developer, and >>>> they are necessary for accessibility. >>>> >>>> 2. Basic additions to the canvas spec can introduce path objects >>>> which can be bound to DOM elements. >>>> >>>> 3. An example of three buttons in a canvas element is shown, >>>> demonstrating the API. >>>> >>>> 4. Some usage notes. >>>> >>>> .................... >>>> >>>> Part 1. Path objects are common in drawing APIs. >>>> >>>> I've worked a little with Richard's proposal, and done a quick review >>>>on >>>> the major drawing APIs out there that >>>> work on the same level of abstraction as Canvas. >>>> >>>> >>>> >>>>http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flas >>>>h/ >>>> display/GraphicsPath.html >>>> >>>> >>>>http://msdn.microsoft.com/en-us/library/system.drawing.drawing2d.pathda >>>>ta >>>> _members(v=vs.85).aspx >>>> >>>> >>>>http://download.oracle.com/javase/7/docs/api/java/awt/geom/PathIterator >>>>.h >>>> tml >>>> >>>> >>>>http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/ >>>>Co >>>> coaDrawingGuide/Paths/Paths.html >>>> "If you draw the same content repeatedly ... [it] is usually more >>>> efficient to retain an existing NSBezierPath object than to recreate >>>>it >>>> during each drawing cycle." >>>> >>>> That last excerpt applies to a few rendering APIs, not just Apple's >>>> Cocoa. >>>> There's precedent in each of these APIs for a Path data object. >>>> WebGL uses path data by typed arrays. It does not include curves. >>>> SVG has normalized path types: >>>> http://msdn.microsoft.com/en-us/library/ff971968(v=vs.85).aspx >>>> >>>> I do not think we need "retained graphics" in canvas -- but it'd be a >>>> huge boost to developers, >>>> and to accessibility, if we can have a means for sharing path data >>>> between layers. >>>> The HTML map tag is insufficient... We're looking for a replacement. >>>> >>>> Richard has put forward a few proposals, as have I... If we can get >>>> some traction with the group, I'm sure we can add a great feature >>>> to Canvas, making life easier on developers and AT vendors alike. >>>> >>>> Developers have told me that they want mouse events delegated >>>> to the shadow dom, based on paths they've bound. Richard and I >>>> are most concerned with getting path data to ATs; though Richard >>>> has a warmth in his heart for making things easier on developers. >>>> >>>> Many vendors are concerned about mixing retained-mode concepts >>>> with Canvas's immediate mode API. >>>> I hope that by focusing only on static path data, we can move past >>>> that concern. I also hope that by introducing new methods, we can >>>> see improved efficiency and accessibility in Canvas applications. >>>> >>>> Please take the following in good faith, and send the list feedback, >>>> so we can take that feedback, apply it, and push the proposal into >>>> the view of a wider audience. >>>> >>>> Part 2. Creating a CanvasPath object and stringify methods in the >>>>spec. >>>> >>>> I've done a basic supplemental. setElementFromPath binds the current >>>> path onto an element, with an optional zIndex. I've also done a >>>> stringify, >>>> supplemental, which would make life a lot easier on developers when it >>>> comes to getting a string for the current path or transformation >>>>matrix. >>>> >>>> // CanvasPath extension >>>> [Supplemental] >>>> interface CanvasRenderingContext2D { >>>> void beginFromPath(in CanvasPath path) ; >>>> void setElementPath(in Element element, in optional DOMString >>>> zIndex) ; >>>> CanvasPath getCurrentPath(); >>>> } >>>> >>>> interface CanvasPath { >>>> // opaque object >>>> } >>>> >>>> // Stringify extensions, defined by SVG normalized paths and >>>>CSSMatrix. >>>> [Supplemental] >>>> interface CanvasRenderingContext2D { >>>> void beginFromPath(in normalizedPathSegList path) ; >>>> void beginFromPath(in DOMString path) ; >>>> CSSMatrix getCurrentMatrix(); >>>> void setCurrentMatrix(in CSSMatrix matrix); >>>> void setCurrentMatrix(in DOMString matrix); >>>> } >>>> [Supplemental] >>>> interface CanvasPath { >>>> stringifier; // returns normalized path >>>> } >>>> >>>> >>>> Part 3. Example of an accessible Canvas, keyboard and pointer device. >>>> /* >>>> Draw three buttons, 100x100 pixels each, spaced 40px apart >>>> If a button is in focus, set its zIndex higher than the other >>>> buttons, >>>> in case of overlap if spacing is changed. >>>> */ >>>> >>>> var canvas = document.getElementById(ŒcanvasTest¹); canvas.width += 0; >>>> // reset state. >>>> var ctx = canvas.getContext(¹2d¹); >>>> var buttons = { 'a': 'Button A', 'b': 'Button B','c': 'Button C' } >>>> var width = 100; var height=30; var padding = 40; >>>> ctx.font = '20px sans-serif'; >>>> ctx.beginPath(); >>>> ctx.rect(0,0,width,height); >>>> var path = ctx.getCurrentPath(); >>>> for(var i in buttons) { >>>> var button = document.getElementById('button_'+i); >>>> if(!button) { >>>> button = document.createElement('input'); >>>> button.setAttribute('id', 'button_'+i); >>>> button.setAttribute('name', i); >>>> button.setAttribute('value', buttons[i]); >>>> button.setAttribute('type','submit'); >>>> if(0) button.onclick = function() { alert("It works!"); }; >>>> if(0) button.onfocus = function() { >>>> if(button.getAttribute('data-myeasypathdata')) { >>>> button.parentNode.getContext('2d').beginFromPath( >>>> button.getAttribute('data-myeasypathdata') ); >>>> >>>> button.parentNode.getContext('2d').drawFocusRing(button); >>>> } >>>> }; >>>> canvas.appendChild(button); >>>> } >>>> var isFocused = document.activeElement == button; >>>> var zIndex = isFocused ? "1" : "0"; >>>> ctx.beginFromPath(path); >>>> ctx.setElementPath(button, zIndex); >>>> if(isFocused) ctx.drawFocusRing(button); >>>> if(0) button.setAttribute('data-myeasypathdata', >>>> ctx.getCurrentPath()); >>>> ctx.fillStyle = 'black'; >>>> ctx.fill(); >>>> ctx.fillStyle = 'white'; >>>> var textBaseline = ....; >>>> ctx.fillText(buttons[i], ( width - >>>>ctx.measureText(buttons[i]).width >>>> ) / 2, textBaseline); >>>> ctx.translate(width + padding, 0); >>>> } >>>> >>>> >>>> Part 4. Usage notes. >>>> >>>> Stringify options allow developers more room for loose coupling of >>>> events; >>>> they also allow better interop between SVG and Canvas... They help >>>>with >>>> debugging; >>>> they can reduce the overhead of complex objects, by reducing the >>>>number >>>> of method >>>> calls, considerably. >>>> >>>> z-index is necessary for coordinate/pointer-based interfaces; things >>>> overlap. >>>> >>>> Transformations will be used heavily in animations. Transformations >>>>are >>>> live, in canvas. >>>> Notice that only one translate method is called, incrementally moving >>>> the origin along the x axis. >>>> >>>> >>>> ----- >>>> >>>> >>>> There are several ATs which require coordinates / paths ahead of time. >>>> Some ATs get by this issue by focusing on an element before requesting >>>> the bounding box. >>>> >>>> ViewPlus produces the "IVEO Hands-on-Learning System": >>>> http://www.viewplus.com/ >>>> >>>> Apple produces Mobile Safari, with "VoiceOver": >>>> http://www.apple.com/accessibility/iphone/vision.html >>>> >>>> Both of these products weighed heavily in the design of the proposal. >>>> >>>> Input is appreciated. >>>> >>>> -Charles >>>> >>>> >>>> On 6/20/2011 3:45 PM, Richard Schwerdtfeger wrote: >>>> >>>> I have some concerns about this. It means: >>>> >>>> - The author has to do the transformations >>>> - We have to add DOM attributes >>>> - The mouse events are routed to canvas and not the sub-DOM elements >>>> where the keyboard handling is going on. The user agent could process >>>> the mouse events at canvas and then propagate them to the >>>>corresponding >>>> DOM object in the subtree. >>>> >>>> Your proposal does have the advantage that the bounds of the objects >>>>are >>>> in the DOM but for HTML we don't have this for any of the DOM elements >>>> now. >>>> >>>> What I was thinking was the following: >>>> >>>> - Today Canvas has the notion of a context >>>> - We allow the author to have the same context (with methods) for each >>>> drawing object and apply a bounds and z-index as you suggest. >>>> - We then bind each drawing object to the canvas subtree DOM element: >>>> >>>> So each drawing object would be an instance of a canvas context with >>>> methods were we do something like: >>>> >>>> 1. we assume that the canvas element when the page is created is an >>>> instance of a canvasObject (having a context) >>>> 2. we assume that drawingOjects are a subclass of canvasObject that >>>> support all the canvas2DAPI in canvasObject with some additions such >>>>as: >>>> - ZIndex attribute >>>> - a bounding drawing path and methods for modifying them >>>> - a method for associating the drawingObject with a canvas subtree DOM >>>> element. >>>> 3. we add an method to canvas that says addDrawObject. >>>> On the canvas element we have the following: >>>> >>>> var canvas = document.getElementById(ŒcanvasTest¹); >>>> if (canvas.getContext){ >>>> var ctx = canvas.getContext(¹2d¹); >>>> DO = new drawingObject(); >>>> dctx= DO.getContext('2d'); >>>> dctx.ZIndex="2"; >>>> dctx.beginPath(); >>>> dctx.moveTo(130,100); >>>> dctx.lineTo... >>>> ... >>>> dctx.setPathtoBounds(); >>>> dctx.setDOMSubtreeNode(foo); //Foo is a subtree node where keyboard >>>> events go to and we do our accessibility enablement to populate the >>>> accessibility tree >>>> //Internally the user agents maps the bounding rectangle for the >>>> accessible of the DOM object as a best fit rectangle of the >>>> //path used to form the bounds of the drawing object >>>> ctx.addDrawingObject(DO); >>>> } >>>> >>>> Now the user agent the information needed to do the hit testing and >>>> retained mode graphics (I am oversimplifying for illustration >>>>purposes) >>>> to be able to track the pointing device input and routing it to the >>>>same >>>> DOM objects that process the keyboard and all the other accessibility >>>> information. This includes hit testing. >>>> >>>> Mike provided feedback on the HTML A11Y call that authors did not want >>>> to do the hit testing themselves. >>>> Rich Schwerdtfeger >>>> CTO Accessibility Software Group >>>> >>>> [cid:part1.01000009.05010600@jumis.com]Frank Olivier ---06/20/2011 >>>> 11:01:54 AM---I would leave hit testing up to the (javascript) >>>>author. I >>>> would recommend that they set existing x, >>>> >>>> From: Frank Olivier >>>> <Frank.Olivier@microsoft.com><mailto:Frank.Olivier@microsoft.com> >>>> To: Richard Schwerdtfeger/Austin/IBM@IBMUS, >>>> "chuck@jumis.com"<mailto:chuck@jumis.com> >>>> <chuck@jumis.com><mailto:chuck@jumis.com>, >>>> "Mike@w3.org"<mailto:Mike@w3.org> <Mike@w3.org><mailto:Mike@w3.org>, >>>> "david.bolter@gmail.com"<mailto:david.bolter@gmail.com> >>>> <david.bolter@gmail.com><mailto:david.bolter@gmail.com>, Cynthia >>>>Shelly >>>> <cyns@microsoft.com><mailto:cyns@microsoft.com> >>>> Cc: "public-canvas-api@w3.org"<mailto:public-canvas-api@w3.org> >>>> <public-canvas-api@w3.org><mailto:public-canvas-api@w3.org>, >>>> "public-html-a11y@w3.org"<mailto:public-html-a11y@w3.org> >>>> <public-html-a11y@w3.org><mailto:public-html-a11y@w3.org>, >>>> "public-html@w3.org"<mailto:public-html@w3.org> >>>> <public-html@w3.org><mailto:public-html@w3.org> >>>> Date: 06/20/2011 11:01 AM >>>> Subject: RE: hit testing and retained graphics >>>> Sent by: public-html-request@w3.org<mailto:public-html-request@w3.org> >>>> >>>> ________________________________ >>>> >>>> >>>> >>>> I would leave hit testing up to the (javascript) author. I would >>>> recommend that they set existing x,y position, z-index attributes on >>>>the >>>> DOM objects in the canvas subtree to report what the UI 'looks like' >>>>to >>>> AT tools. This way, the AT tools don't need to change - this part of >>>>the >>>> DOM is no different to them than any other part - and authors need to >>>>be >>>> annotating canvas DOM objects with correct information anyway (labels, >>>> aria attributes, etc). >>>> >>>> From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com] >>>> Sent: Friday, June 17, 2011 11:42 AM >>>> To: chuck@jumis.com<mailto:chuck@jumis.com>; Frank Olivier; >>>> Mike@w3.org<mailto:Mike@w3.org>; >>>> david.bolter@gmail.com<mailto:david.bolter@gmail.com>; Cynthia Shelly >>>> Cc: public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>; >>>> public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>; >>>> public-html@w3.org<mailto:public-html@w3.org> >>>> Subject: hit testing and retained graphics >>>> >>>> Charles, Frank, Mike, >>>> >>>> I am back from vacation. How far do we need to go with hit testing? >>>> Right now I am looking at associating a closed draw path with a DOM >>>> object in the canvas subtree. We would then need to address the >>>>routing >>>> of pointing device input events to the DOM object. The drawing path >>>>can >>>> be used to provide bound information to platform accessibility API. >>>> >>>> Do we need to bind any other drawing properties to the canvas object - >>>> similar to the way device context's are handled on graphic subsystems >>>> like Windows? >>>> >>>> Mike, I am including you as before I went on vacation you indicated >>>>that >>>> a number of developers desired this feature and wanted to be involved. >>>> >>>> Rich >>>> >>>> >>>> Rich Schwerdtfeger >>>> CTO Accessibility Software Group >>>> >>>> >>>> >>> >>> >>> -- >>> Charles McCathieNevile Opera Software, Standards Group >>> je parle français -- hablo español -- jeg lærer norsk >>> http://my.opera.com/chaals Try Opera: http://www.opera.com >>> >> >>
Received on Thursday, 23 June 2011 07:38:28 UTC