Re: hit testing and retained graphics

Hi everyone,

We at Zynga use hit testing to a great extend in our isometric games and have long searched for a solution that removes some of the processing burden from us. I therefore very much disagree that hit testing should be up to the js developer. If possible, this is a perfect candidate for a job the browser can help us with. And by "us", I am probably talking about almost every JS game developer.

I proposed an extension to CSS in April that a lot of folks I talked to liked, as it is both simple and effective for simple hit testing on DOM elements. You can read the summary in this thread: http://lists.w3.org/Archives/Public/www-style/2011Apr/0558.html

Paul

Von: Charles Pritchard <chuck@jumis.com<mailto:chuck@jumis.com>>
Datum: Tue, 21 Jun 2011 21:36:57 -0700
An: Richard Schwerdtfeger <schwer@us.ibm.com<mailto:schwer@us.ibm.com>>
Cc: Frank Olivier <Frank.Olivier@microsoft.com<mailto:Frank.Olivier@microsoft.com>>, Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>, "david.bolter@gmail.com<mailto:david.bolter@gmail.com>" <david.bolter@gmail.com<mailto:david.bolter@gmail.com>>, "Mike@w3.org<mailto:Mike@w3.org>" <Mike@w3.org<mailto:Mike@w3.org>>, "public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>" <public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>>, "public-html@w3.org<mailto:public-html@w3.org>" <public-html@w3.org<mailto:public-html@w3.org>>, "public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>" <public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>>, "public-html-request@w3.org<mailto:public-html-request@w3.org>" <public-html-request@w3.org<mailto:public-html-request@w3.org>>
Betreff: Re: hit testing and retained graphics

This is a lengthy e-mail. Summary; three parts:

1. Retained Path objects are ubiquitous in basic 2d drawing APIs;
they are efficient, they are managed by the developer, and
they are necessary for accessibility.

2. Basic additions to the canvas spec can introduce path objects
which can be bound to DOM elements.

3. An example of three buttons in a canvas element is shown,
demonstrating the API.

4. Some usage notes.

....................

Part 1. Path objects are common in drawing APIs.

I've worked a little with Richard's proposal, and done a quick review on the major drawing APIs out there that
work on the same level of abstraction as Canvas.

http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display/GraphicsPath.html
http://msdn.microsoft.com/en-us/library/system.drawing.drawing2d.pathdata_members(v=vs.85).aspx
http://download.oracle.com/javase/7/docs/api/java/awt/geom/PathIterator.html
http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CocoaDrawingGuide/Paths/Paths.html
"If you draw the same content repeatedly ... [it] is usually more efficient to retain an existing NSBezierPath object than to recreate it during each drawing cycle."

That last excerpt applies to a few rendering APIs, not just Apple's Cocoa.
There's precedent in each of these APIs for a Path data object.
WebGL uses path data by typed arrays. It does not include curves.
SVG has normalized path types:
http://msdn.microsoft.com/en-us/library/ff971968(v=vs.85).aspx

I do not think we need "retained graphics" in canvas -- but it'd be a huge boost to developers,
and to accessibility, if we can have a means for sharing path data between layers.
The HTML map tag is insufficient... We're looking for a replacement.

Richard has put forward a few proposals, as have I... If we can get
some traction with the group, I'm sure we can add a great feature
to Canvas, making life easier on developers and AT vendors alike.

Developers have told me that they want mouse events delegated
to the shadow dom, based on paths they've bound. Richard and I
are most concerned with getting path data to ATs; though Richard
has a warmth in his heart for making things easier on developers.

Many vendors are concerned about mixing retained-mode concepts
with Canvas's immediate mode API.
I hope that by focusing only on static path data, we can move past
that concern. I also hope that by introducing new methods, we can
see improved efficiency and accessibility in Canvas applications.

Please take the following in good faith, and send the list feedback,
so we can take that feedback, apply it, and push the proposal into
the view of a wider audience.

Part 2. Creating a CanvasPath object and stringify methods in the spec.

I've done a basic supplemental. setElementFromPath binds the current
path onto an element, with an optional zIndex. I've also done a stringify,
supplemental, which would make life a lot easier on developers when it
comes to getting a string for the current path or transformation matrix.

// CanvasPath extension
[Supplemental]
interface CanvasRenderingContext2D {
    void beginFromPath(in CanvasPath path) ;
    void setElementPath(in Element element, in optional DOMString zIndex) ;
    CanvasPath getCurrentPath();
}

interface CanvasPath {
    // opaque object
}

// Stringify extensions, defined by SVG normalized paths and CSSMatrix.
[Supplemental]
interface CanvasRenderingContext2D {
    void beginFromPath(in normalizedPathSegList path) ;
    void beginFromPath(in DOMString path) ;
    CSSMatrix getCurrentMatrix();
    void setCurrentMatrix(in CSSMatrix matrix);
    void setCurrentMatrix(in DOMString matrix);
}
[Supplemental]
interface CanvasPath {
    stringifier; // returns normalized path
}


Part 3. Example of an accessible Canvas, keyboard and pointer device.
/*
 Draw three buttons, 100x100 pixels each, spaced 40px apart
    If a button is in focus, set its zIndex higher than the other buttons,
    in case of overlap if spacing is changed.
*/

var canvas = document.getElementById(‘canvasTest’); canvas.width += 0; // reset state.
var ctx = canvas.getContext(’2d’);
var buttons = { 'a': 'Button A', 'b': 'Button B','c': 'Button C' }
var width = 100; var height=30; var padding = 40;
ctx.font = '20px sans-serif';
ctx.beginPath();
ctx.rect(0,0,width,height);
var path = ctx.getCurrentPath();
for(var i in buttons) {
    var button = document.getElementById('button_'+i);
    if(!button) {
        button = document.createElement('input');
        button.setAttribute('id', 'button_'+i);
        button.setAttribute('name', i);
        button.setAttribute('value', buttons[i]);
        button.setAttribute('type','submit');
        if(0) button.onclick = function() { alert("It works!"); };
        if(0) button.onfocus = function() {
            if(button.getAttribute('data-myeasypathdata')) {
                button.parentNode.getContext('2d').beginFromPath( button.getAttribute('data-myeasypathdata') );
                button.parentNode.getContext('2d').drawFocusRing(button);
            }
        };
        canvas.appendChild(button);
    }
    var isFocused = document.activeElement == button;
    var zIndex = isFocused ? "1" : "0";
    ctx.beginFromPath(path);
    ctx.setElementPath(button, zIndex);
    if(isFocused) ctx.drawFocusRing(button);
    if(0) button.setAttribute('data-myeasypathdata', ctx.getCurrentPath());
    ctx.fillStyle = 'black';
    ctx.fill();
    ctx.fillStyle = 'white';
    var textBaseline = ....;
    ctx.fillText(buttons[i], ( width - ctx.measureText(buttons[i]).width ) / 2, textBaseline);
    ctx.translate(width + padding, 0);
}


Part 4. Usage notes.

Stringify options allow developers more room for loose coupling of events;
they also allow better interop between SVG and Canvas... They help with debugging;
they can reduce the overhead of complex objects, by reducing the number of method
calls, considerably.

z-index is necessary for coordinate/pointer-based interfaces; things overlap.

Transformations will be used heavily in animations. Transformations are live, in canvas.
Notice that only one translate method is called, incrementally moving the origin along the x axis.


-----


There are several ATs which require coordinates / paths ahead of time.
Some ATs get by this issue by focusing on an element before requesting the bounding box.

ViewPlus produces the "IVEO Hands-on-Learning System":
http://www.viewplus.com/

Apple produces Mobile Safari, with "VoiceOver":
http://www.apple.com/accessibility/iphone/vision.html

Both of these products weighed heavily in the design of the proposal.

Input is appreciated.

-Charles


On 6/20/2011 3:45 PM, Richard Schwerdtfeger wrote:

I have some concerns about this. It means:

- The author has to do the transformations
- We have to add DOM attributes
- The mouse events are routed to canvas and not the sub-DOM elements where the keyboard handling is going on. The user agent could process the mouse events at canvas and then propagate them to the corresponding DOM object in the subtree.

Your proposal does have the advantage that the bounds of the objects are in the DOM but for HTML we don't have this for any of the DOM elements now.

What I was thinking was the following:

- Today Canvas has the notion of a context
- We allow the author to have the same context (with methods) for each drawing object and apply a bounds and z-index as you suggest.
- We then bind each drawing object to the canvas subtree DOM element:

So each drawing object would be an instance of a canvas context with methods were we do something like:

1. we assume that the canvas element when the page is created is an instance of a canvasObject (having a context)
2. we assume that drawingOjects are a subclass of canvasObject that support all the canvas2DAPI in canvasObject with some additions such as:
- ZIndex attribute
- a bounding drawing path and methods for modifying them
- a method for associating the drawingObject with a canvas subtree DOM element.
3. we add an method to canvas that says addDrawObject.
On the canvas element we have the following:

var canvas = document.getElementById(‘canvasTest’);
if (canvas.getContext){
var ctx = canvas.getContext(’2d’);
DO = new drawingObject();
dctx= DO.getContext('2d');
dctx.ZIndex="2";
dctx.beginPath();
dctx.moveTo(130,100);
dctx.lineTo...
...
dctx.setPathtoBounds();
dctx.setDOMSubtreeNode(foo); //Foo is a subtree node where keyboard events go to and we do our accessibility enablement to populate the accessibility tree
//Internally the user agents maps the bounding rectangle for the accessible of the DOM object as a best fit rectangle of the
//path used to form the bounds of the drawing object
ctx.addDrawingObject(DO);
}

Now the user agent the information needed to do the hit testing and retained mode graphics (I am oversimplifying for illustration purposes) to be able to track the pointing device input and routing it to the same DOM objects that process the keyboard and all the other accessibility information. This includes hit testing.

Mike provided feedback on the HTML A11Y call that authors did not want to do the hit testing themselves.
Rich Schwerdtfeger
CTO Accessibility Software Group

[cid:part1.01000009.05010600@jumis.com]Frank Olivier ---06/20/2011 11:01:54 AM---I would leave hit testing up to the (javascript) author. I would recommend that they set existing x,

From: Frank Olivier <Frank.Olivier@microsoft.com><mailto:Frank.Olivier@microsoft.com>
To: Richard Schwerdtfeger/Austin/IBM@IBMUS, "chuck@jumis.com"<mailto:chuck@jumis.com> <chuck@jumis.com><mailto:chuck@jumis.com>, "Mike@w3.org"<mailto:Mike@w3.org> <Mike@w3.org><mailto:Mike@w3.org>, "david.bolter@gmail.com"<mailto:david.bolter@gmail.com> <david.bolter@gmail.com><mailto:david.bolter@gmail.com>, Cynthia Shelly <cyns@microsoft.com><mailto:cyns@microsoft.com>
Cc: "public-canvas-api@w3.org"<mailto:public-canvas-api@w3.org> <public-canvas-api@w3.org><mailto:public-canvas-api@w3.org>, "public-html-a11y@w3.org"<mailto:public-html-a11y@w3.org> <public-html-a11y@w3.org><mailto:public-html-a11y@w3.org>, "public-html@w3.org"<mailto:public-html@w3.org> <public-html@w3.org><mailto:public-html@w3.org>
Date: 06/20/2011 11:01 AM
Subject: RE: hit testing and retained graphics
Sent by: public-html-request@w3.org<mailto:public-html-request@w3.org>

________________________________



I would leave hit testing up to the (javascript) author. I would recommend that they set existing x,y position, z-index attributes on the DOM objects in the canvas subtree to report what the UI 'looks like' to AT tools. This way, the AT tools don't need to change - this part of the DOM is no different to them than any other part - and authors need to be annotating canvas DOM objects with correct information anyway (labels, aria attributes, etc).

From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
Sent: Friday, June 17, 2011 11:42 AM
To: chuck@jumis.com<mailto:chuck@jumis.com>; Frank Olivier; Mike@w3.org<mailto:Mike@w3.org>; david.bolter@gmail.com<mailto:david.bolter@gmail.com>; Cynthia Shelly
Cc: public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>; public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>; public-html@w3.org<mailto:public-html@w3.org>
Subject: hit testing and retained graphics

Charles, Frank, Mike,

I am back from vacation. How far do we need to go with hit testing? Right now I am looking at associating a closed draw path with a DOM object in the canvas subtree. We would then need to address the routing of pointing device input events to the DOM object. The drawing path can be used to provide bound information to platform accessibility API.

Do we need to bind any other drawing properties to the canvas object - similar to the way device context's are handled on graphic subsystems like Windows?

Mike, I am including you as before I went on vacation you indicated that a number of developers desired this feature and wanted to be involved.

Rich


Rich Schwerdtfeger
CTO Accessibility Software Group