RE: draft of hit testing on active regions (actually paths) in canvas

A preview of that draft, based on notes I took during the meeting:

These new methods manage the association of pixels drawn with canvas fallback elements:
beginElement(Element element);
endElement(Element element);
eraseElement(element) ;
eraseElements();

So, code example:

For a html page with this content:
<canvas>
  <input type=checkbox id=checkboxA >Checkbox A</input>
  <input type=checkbox id=checkboxB >Checkbox B</input>
</canvas>

By default, all pixels on a canvas are 'unassociated'. The author would use these new methods to associate pixels with fallback elements.

When drawing the checkbox:
// Rendering code
ctx.beginElement(checkboxA);     //Tell the canvas context that all non-fully-transparent pixels that we are rendering from this point on represents the clickable area for the checkbox UI
// Draw a background for the checkbox
ctx.fillStyle = 'white';
ctx.rect(0, 0, 100, 20);
context.fill();

// Draw the actual checkbox
ctx.rect(5, 5, 10, 10);
ctx.stroke();

if (checkboxA.checked)
{
  ctx.fillStyle = 'black';
  ctx.fill();
}
ctx.fillText('checkbox', 20, 10);
endElement(checkboxA); //We are done with element rendering

The UA/magnifier would associate any pixels rendered from beginElement() to endElement() with the fallback dom element; the fallback dom element 'owns' those pixels on the screen. To resolve overlap, last write wins. (Implementation note: A UA could use a bitmap to associate pixels with fallback dom elements. A value of 0 in that bitmap could mean 'no association'; value 1 could mean 'first element in the fallback dom element lookup table', etc.)

Between beginElement() and endElement(), the UA would keep track of the extent of the pixels that are touched by drawing operations; it would use this to compute the bounding rectangle when a magnifier asks for the element's extent.

At some point in the future, when my canvas bitmap no longer has that checkbox visible:

               ctx.eraseElement(checkboxA); //Clear all pixels 'owned' by checkboxA
               (This imo assumes that you are removing or disabling the corresponding DOM checkbox input element)

All the pixels that were associated with checkboxA would now be unassociated with any fallback dom element.

eraseElements() does the same, but for all fallback dom elements.

I could (potentially) also clear some of the pixels that are associated with a corresponding canvas element:
ctx.beginElement(null);   //Tell the canvas context that all non-fully-transparent pixels that we are rendering from this point on should not be part of any element
ctx.fillStyle = 'white';
ctx.rect(0, 0, 100, 100);
context.fill();
endElement(checkboxA); //We are done with element rendering




From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
Sent: Monday, October 31, 2011 4:32 PM
To: Jonas Sicking; public-canvas-api@w3.org; public-html-a11y@w3.org
Cc: chucko@jumis.com; Cynthia Shelly; david.bolter@gmail.com; Frank Olivier; janina@rednote.net; jbrewer@w3.org
Subject: Re: draft of hit testing on active regions (actually paths) in canvas


Jonas,

We discussed your points on the PF call today. Apple was present. I will write up a revised draft that associated drawing to the elements per your request.

Thanks for you feedback.

Rich


Rich Schwerdtfeger
CTO Accessibility Software Group

[Inactive hide details for Richard Schwerdtfeger---10/31/2011 11:03:00 AM---Rich Schwerdtfeger CTO Accessibility Software Group]Richard Schwerdtfeger---10/31/2011 11:03:00 AM---Rich Schwerdtfeger CTO Accessibility Software Group

From: Richard Schwerdtfeger/Austin/IBM@IBMUS
To: Jonas Sicking <jonas@sicking.cc<mailto:jonas@sicking.cc>>,
Cc: chucko@jumis.com<mailto:chucko@jumis.com>, cyns@exchange.microsoft.com<mailto:cyns@exchange.microsoft.com>, david.bolter@gmail.com<mailto:david.bolter@gmail.com>, franko@microsoft.com<mailto:franko@microsoft.com>, janina@rednote.net<mailto:janina@rednote.net>, jbrewer@w3.org<mailto:jbrewer@w3.org>, public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>, public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>
Date: 10/31/2011 11:03 AM
Subject: Re: draft of hit testing on active regions (actually paths) in canvas

________________________________





Rich Schwerdtfeger
CTO Accessibility Software Group

Jonas Sicking <jonas@sicking.cc<mailto:jonas@sicking.cc>> wrote on 10/27/2011 07:42:40 PM:

> From: Jonas Sicking <jonas@sicking.cc<mailto:jonas@sicking.cc>>
> To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
> Cc: franko@microsoft.com<mailto:franko@microsoft.com>, chucko@jumis.com<mailto:chucko@jumis.com>, david.bolter@gmail.com<mailto:david.bolter@gmail.com>,
> cyns@exchange.microsoft.com<mailto:cyns@exchange.microsoft.com>, public-html-a11y@w3.org<mailto:public-html-a11y@w3.org>,
> janina@rednote.net<mailto:janina@rednote.net>, jbrewer@w3.org<mailto:jbrewer@w3.org>, public-canvas-api@w3.org<mailto:public-canvas-api@w3.org>
> Date: 10/27/2011 07:43 PM
> Subject: Re: draft of hit testing on active regions (actually paths)in canvas
>
> Some feedback:
>
> Limiting to only allowing paths seems like a unfortunate limitation.
> For example it misses calls to drawImage which I think is quite
> common. I'd rather prefer a call where you say "all drawing operations
> from this point should be considered drawing for element X", then let
> the page do arbitrary drawing operations, then have a second call
> which says "i'm now drawing for element Y" or "I'm now drawing for no
> element".
>

I see. So what would you use compute the bounds? Do you use the all the points to compute a single rectangle for hit testing? ... or do you compute multiple paths for an element to do hit testing on?


> What is the use case for the zIndex argument? The actual pixel drawing
> operations hasn't had a need for that so far and instead rely on the
> painters algorithm. It seems better to me to have a direct mapping
> between the drawing operations and the accessibility API.
>
zIndex: Performance and authors may have overlapping drawing objects.

> Why return false rather than throw an exception if the element doesn't
> exist? Also what do you mean by "doesn't exist". If the element
> doesn't exist then how could the page have a reference to it and pass
> that reference to the function? Do you mean "element isn't inside the
> canvas"?
>
Throwing an exception makes sense. I meant the element did not exist. I am thinking that the author may have content outside fallback such as a drop down menu. We could limit the elements to the fallback content. I would like your feedback on that.

> In general I think I prefer the API I proposed in [1]. Can you
> describe what problems you were trying to solve with your changes
> compared to that proposal?
>
> [1] http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html
>
> / Jonas
>
> On Thu, Oct 27, 2011 at 4:56 PM, Richard Schwerdtfeger
> <schwer@us.ibm.com<mailto:schwer@us.ibm.com>> wrote:
> > Hi Frank,
> >
> > Thank you for the call today. I am glad we are on the same page onthe three
> > basic methods for hit testing. I took the basic methods that Jonas had put
> > out earlier on the list and the discussion we had in San Diego and put
> > together an active regions and hit testing section. I also agree with you
> > this is for when we have multiple interactive elements within canvas. I was
> > glad to see that customers are starting to come to Microsoft to talk about
> > building more interactive widgets with canvas - not because I want them to
> > but more because it shows we are making the right moves to address canvas
> > accessibility.
> >
> > Jonas, Charles, Frank, David, Cynthia
> >
> > This is a first draft and the section I would like you to look at is section
> > 16. We can change the names to whatever we want. I have not yet added spec.
> > information as to how existing functions like scrollElementIntoView should
> > work on these elements as they should be included in the layout engine
> > positioning per Jonas earlier feedback. I have also not stated how we would
> > convert the paths to rectangles for the underlying OS platform accessibility
> > APIs. We could either do this in the canvas spec. or in the accessibility
> > API mapping guide that Cynthia is leading. We just need to decide.
> > Personally, I prefer the one stop shopping.
> >
> > I am sure Ian will want to tweak or rewrite the algorithm section processing
> > as he has his own consistent way of doing that but I took my best shot at
> > it.
> >
> > Please review section 16.
> >
> > (See attached file: clickableregion.html)
> >
> > Since, like many of you I am going to TPAC, I will not have time to make
> > changes prior to the face to face. Janina please put a link to this document
> > in your canvas agenda for TPAC.
> >
> > See you all in California.
> >
> >
> > Rich Schwerdtfeger
> >
>