Hit testing use cases Re: Request to re-open issue 131

On Sun, 18 Dec 2011 11:15:59 +0100, Steve Faulkner  
<faulkner.steve@gmail.com> wrote:

> Jonas wrote:
>
>> However even for hit testing and focus management, maybe we would
>> design the API differently if we weren't trying to use them to built
>> text editors. I really don't know enough about accessibility to fully
>> answer that.

One proposal was to re-use the image map paradigm, because it is well  
known, widely implemented interoperably (modulo some implementations being  
partial - just like HTML5 and SVG and CSS3...), and because it already  
provides at least a great many of the necessary hooks to make an  
interactive visual representation of something accessible. There are  
indeed other ways to do this (e.g. we could use the aria syntax that  
builds the same thing, we could design the API from scratch, we could  
adopt something from SVG, etc), but what I do know about accessibility  
leads me to doubt that we need to reinvent this wheel again from scratch.  
We should of course examine it for the use cases which go beyond  
"accessibility" before taking such a decision.

> For hit testing, example uses and background is provided by  Frank
> Olivier in the  doc i pointed to:
> http://www.w3.org/wiki/Canvas_hit_testing
> He has also publicly stated that building text editors in canvas is a
> 'fools errand':
> http://lists.w3.org/Archives/Public/public-html/2011Nov/0210
> So I think it is reasonable to assume that Franks hit testing proposal
> is not motivated by the desire to build text editors in canvas.

I also believe that building text editors in canvas is a fool's errand,  
but that hit testing with a level of "what pixel was touched" would be  
valuable.

There are a lot of use cases. I'll detail some and roughly describe some  
more. I cc'ed Paul because I recall him saying that for games this would  
be really valuable.

Use case: Graphics editors

Canvas has been used to create numerous applications for manipulating  
graphics. Where button controls are drawn into the canvas, some relatively  
simple calculation needs to be done to check whether they have been  
activated. Where there is any dynamic aspect to this (e.g. a brush  
selection button results in a number of possible brushes being shown) the  
complexity of the calculations required increases rapidly. Elements which  
can have a listener applied and whose coordinates in the canvas can be  
easily described (which is already done to draw or "undraw" them) reduce  
the complexity and increase the maintainability of the code.
(Having had such tools on the Apple II Plus (1983) I find the argument  
that this sot of application is not suited to keyboard use a load of  
post-hoc rationalisation of nonsense, although I recognise that it has  
been successfully employed in the past for setting legal requirements. The  
difference between that and technical use cases is one related to where  
the burden is imposed. Applying the principle that things should be  
easiest for users and hardest for proponents of theoretical technical  
purity, I conclude that making this possible in a browser is less burden  
on the world than making every app developer do it on their own).

Use case: Adaptable games

Many very simple graphical games (pitched around kindergarten level) can  
be made in either SVG, HTML+CSS or canvas, with the most important  
determinant being the programming paradigm most suited to the developer's  
further needs. peepo.com provided a suite of such games for people with  
severe intellectual disability, explicitly designed to familiarise people  
with the use of the mouse to interact with on-screen objects which were  
randomly drawn on the canvas. The ability to hit-test and determine which  
of a set of objects is nearly all of the active code in a canvas  
implementation (the examples on peepo happened to use SVG because the  
author was more familiar with it, and it is indeed better suited to this  
kind of application).
NB There are many demos of "piles of photos" which have similar underlying  
code requirements.

Use case: Jigsaw puzzles

Canvas is ideally suited to making dynamic jigsaw puzzles. Taking a  
picture, cutting it into fragments, and randomly distributing them on the  
canvas is much easier in canvas than SVG. But to enable a user to  
dynamically interact with the pieces, some kind of hit-testing is  
required. Doing this with listeners on elements whose location can be  
updated, rather than having to do the calculation in javascript every  
time, makes a significant difference to the amount of code required and to  
its complexity and maintainability.
In many ways this is a specialisation of the previous use case, but one  
where the use of canvas really is far more natural than any alternative.

Use case: Accessible visual testing

The tests required for a driving license in Victoria (and presumably in  
many countries, since the underlying parts are basically the same) include  
many based on visual interpretation of a scene and identifying important  
points. The level of visual perception required for the normal  
presentations of the test (on relatively small touch screens) is notably  
higher than the level of visual perception required to drive safely.  
Similarly, the use of visual interactive canvases to explain chemistry and  
then examine students is decades old*, but a certain proportion of  
prospective chemists will need to zoom and pan around the presentation  
because their vision is insufficient to interact with the default system  
(while perfectly good enough to be expert chemists - something that in  
fact people can do being completely blind). The ability to dynamically  
adjust the size of the canvas without having to re-calculate all hit zones  
simplifies the task of making the system meet legal requirements for  
accessibility.
(I studied chemistry decades ago and had to squint to use such systems.)

Some other places this would be helpful beyond simple generalisation of  
the use cases above include:
+ 2-dimensional interactive map and map-based interactions (from  
treasure-map games to crowd-sourcing information about danger spots in a  
given area);
+ offering the ability to move annotations on a map so they don't overlap;
+ games which intrinsically require zoom and pan e.g. to mimic the limited  
view a pilot has at any time over the entire set of controls available in  
a flight deck;
+ Interactive UI design tools that allow for live addition of  
functionality to components initially drawn and manipulated on a canvas

This is by no means exhaustive, just some illustrative examples.

cheers

Chaals

-- 
Charles 'chaals' McCathieNevile  Opera Software, Standards Group
     je parle français -- hablo español -- jeg kan litt norsk
http://my.opera.com/chaals       Try Opera: http://www.opera.com

Received on Sunday, 18 December 2011 16:45:31 UTC