Re: draft of hit testing on active regions (actually paths) in canvas

On Thu, Oct 27, 2011 at 7:27 PM, Charles Pritchard <chuck@jumis.com> wrote:
> On 10/27/11 5:42 PM, Jonas Sicking wrote:
>>
>> Some feedback:
>>
>> Limiting to only allowing paths seems like a unfortunate limitation.
>> For example it misses calls to drawImage which I think is quite
>> common. I'd rather prefer a call where you say "all drawing operations
>> from this point should be considered drawing for element X", then let
>> the page do arbitrary drawing operations, then have a second call
>> which says "i'm now drawing for element Y" or "I'm now drawing for no
>> element".
>
> These are two very different ideas. I did not realize -how- different your
> proposal was from the one the WG discussed earlier in the year. I'm going to
> try to explain both of them, and please correct me if I've made a mistake.
>
> The setDrawingFor proposal monitors subsequent drawing calls and updates the
> bounding box associated with the target element as drawing calls happen.
>
> // start example
> ctx.setDrawingFor(element);
> ctx.fillRect(0,0,20,20);
> ctx.fillRect(200,200, 100,100);
> ctx.setDrawingFor(null);
> // at end element now has a bounding region from 0,0 to 300,300.
>
> The setPathForElement proposal acts on the current path:
> // start example
> ctx.fillRect(0,0,20,20);
> ctx.fillRect(200,200,100,100);
> ctx.setPathForElement(element);
> // at end element now has a bounding region from 200,200 to 300,300.
>
> Both of these proposals can be implemented with minimal effort in existing
> Canvas code bases.
>
> I'm concerned about supporting non-rectangular regions in large areas.
> http://www.w3.org/Graphics/SVG/WG/track/issues/2421
>
> I'm concerned that the setDrawingFor proposal requires that authors run fill
> or stroke to operate. This would require additional steps for authors
> setting an interactive region:
>
> // Using setDrawingFor without painting
> ctx.save();
> ctx.setDrawingFor(element);
> ctx.fillStyle = 'rgba(0,0,0,0)';
> ctx.fillRect(0,0,20,20);
> ctx.setDrawingFor(null);
> ctx.restore();

Why are the extra .save() and .restore() calls needed here? What would
go wrong if they were left out?

Please note that I'm not an expert in the specifics of canvas drawing
operations.

> // Using setPathForElement without painting
> ctx.beginPath();
> ctx.rect(0,0,20,20);
> ctx.setPathForElement(element);
>
>
>> What is the use case for the zIndex argument? The actual pixel drawing
>> operations hasn't had a need for that so far and instead rely on the
>> painters algorithm. It seems better to me to have a direct mapping
>> between the drawing operations and the accessibility API.
>
> It may be unnecessary. I'll do some deep thinking.
> As you said, authors can typically manage these things.

They already are if they are using canvas, no?

>> In general I think I prefer the API I proposed in [1]. Can you
>> describe what problems you were trying to solve with your changes
>> compared to that proposal?
>>
>> [1]
>> http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html
>
> My primary concern here is path fidelity. Apart from the extra steps I
> outlined earlier, would a mature setDrawingFor implementation keep all of
> the paths for accessibility?
>
> setPathForElement instructs the implementation to maintain the already
> constructed path on some kind of stack or other ordered object.
>
> setDrawingFor would -possibly- maintain a series of paths opposed to a
> series of subpaths.
>
> Basic implementations are most likely to just keep bounding box information
> in either case. More advanced implementations will maintain some kind of
> additional information about the interactive region.

I don't think keeping just bounding box information should be
permitted. My idea is that when the user clicks a <canvas> which has
used the setDrawingFor API, then we forward the click to the actual
elements inside the canvas. Similar to how labels work. I.e. any event
handlers registered on the element would fire. If the element is a
<input type=checkbox> then it would flip its "checkedness" etc. But
only pixels drawn while setDrawingFor is in effect acts as a
forwarding area.

This way authors have a much greater incentive to use the
setDrawingFor API. I.e. they would get much simpler event handling for
all their users. The fact that we'd be able to use the same
information to drive screen magnifiers etc is "just" a convenient side
effect from the authors point of view. This is the type of API which
has worked the best historically. I.e. bolt-on solutions which are
there only for a11y users has seen much less usage than APIs which add
enough semantics to help all users. (This is why we prefer if people
use HTML based UIs, rather than canvas based ones after all, right?)
(Note, i'm not saying that bolt-on is bad or should be avoided. But
when we can find better solutions, I think we should use them).

Similarly, this would act as a way for people to more easily add
different tooltips for different parts of a canvas, or to get a
appropriate mouse cursor when the user is hovering areas of a canvas
which draws pixels corresponding to a <a> anchor.

> These are two very different methods, I like them both. setPathForElement is
> less expensive to implement; less lines of code changed. setDrawingFor could
> trap -more- information, relating more to SVG interop than to a11y.
>
> setPathForElement would simply use the current path, add it to a stack, and
> that stack would be accessible by the UA accessibility APIs. It's feasible
> that the path would be serialized and sent to a supporting AT. Otherwise,
> it'd simply be used for its bounding box information. This requires no
> changes in existing Canvas methods, only the addition of new methods.
>
> setDrawingFor would require hooks to be added to most drawing methods. It
> may require additional logic for strokeWidth. An advanced implementation may
> collect drawing calls while setDrawingFor is running, serializing them into
> SVG and adding them into a Component DOM. This would be a lot of extra work
> on the CPU. At this point super-computers fit in our pockets. It'd be
> reasonable that toDataURL('image/svg+xml') would return a scene-graph from
> the Component DOM.

The fact that it requires hooks in all drawing methods is a feature
IMHO. It ensures that it's easy for the developer to make *all* pixels
drawn for an element clickable, no matter if the pixels are drawn
using a path-based API, or some other API.

I think we'll see significantly more usage if all the author needs to
do is to sprinkle setDrawingFor calls over the code to mark which
drawing calls are made for which elements. If setPathForElements
require authors to create new paths specifically just to mark bounding
boxes then I think fewer people will use it.

/ Jonas

Received on Friday, 28 October 2011 07:01:12 UTC