RE: draft of hit testing on active regions (actually paths) in canvas

Thanks Frank, I will write up a mod. with the new API for us to poke at.

Rich Schwerdtfeger
CTO Accessibility Software Group



From: Frank Olivier <Frank.Olivier@microsoft.com>
To: Richard Schwerdtfeger/Austin/IBM@IBMUS, Jonas Sicking
            <jonas@sicking.cc>, "public-canvas-api@w3.org"
            <public-canvas-api@w3.org>, "public-html-a11y@w3.org"
            <public-html-a11y@w3.org>,
Cc: "chucko@jumis.com" <chucko@jumis.com>, Cynthia Shelly
            <cyns@microsoft.com>, "david.bolter@gmail.com"
            <david.bolter@gmail.com>, "janina@rednote.net"
            <janina@rednote.net>, "jbrewer@w3.org" <jbrewer@w3.org>
Date: 10/31/2011 06:47 PM
Subject: RE: draft of hit testing on active regions (actually paths) in
            canvas



A preview of that draft, based on notes I took during the meeting:

These new methods manage the association of pixels drawn with canvas
fallback elements:
beginElement(Element element);
endElement(Element element);
eraseElement(element) ;
eraseElements();

So, code example:

For a html page with this content:
<canvas>
  <input type=checkbox id=checkboxA >Checkbox A</input>
  <input type=checkbox id=checkboxB >Checkbox B</input>
</canvas>

By default, all pixels on a canvas are ‘unassociated’. The author would use
these new methods to associate pixels with fallback elements.

When drawing the checkbox:
// Rendering code
ctx.beginElement(checkboxA);     //Tell the canvas context that all
non-fully-transparent pixels that we are rendering from this point on
represents the clickable area for the checkbox UI
// Draw a background for the checkbox
ctx.fillStyle = 'white’;
ctx.rect(0, 0, 100, 20);
context.fill();

// Draw the actual checkbox
ctx.rect(5, 5, 10, 10);
ctx.stroke();

if (checkboxA.checked)
{
  ctx.fillStyle = 'black';
  ctx.fill();
}
ctx.fillText(‘checkbox’, 20, 10);
endElement(checkboxA); //We are done with element rendering

The UA/magnifier would associate any pixels rendered from beginElement() to
endElement() with the fallback dom element; the fallback dom element ‘owns’
those pixels on the screen. To resolve overlap, last write wins.
(Implementation note: A UA could use a bitmap to associate pixels with
fallback dom elements. A value of 0 in that bitmap could mean ‘no
association’; value 1 could mean ‘first element in the fallback dom element
lookup table’, etc.)

Between beginElement() and endElement(), the UA would keep track of the
extent of the pixels that are touched by drawing operations; it would use
this to compute the bounding rectangle when a magnifier asks for the
element’s extent.

At some point in the future, when my canvas bitmap no longer has that
checkbox visible:

               ctx.eraseElement(checkboxA); //Clear all pixels ‘owned’ by
checkboxA
               (This imo assumes that you are removing or disabling the
corresponding DOM checkbox input element)

All the pixels that were associated with checkboxA would now be
unassociated with any fallback dom element.

eraseElements() does the same, but for all fallback dom elements.

I could (potentially) also clear some of the pixels that are associated
with a corresponding canvas element:
ctx.beginElement(null);   //Tell the canvas context that all
non-fully-transparent pixels that we are rendering from this point on
should not be part of any element
ctx.fillStyle = 'white’;
ctx.rect(0, 0, 100, 100);
context.fill();
endElement(checkboxA); //We are done with element rendering




From: Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
Sent: Monday, October 31, 2011 4:32 PM
To: Jonas Sicking; public-canvas-api@w3.org; public-html-a11y@w3.org
Cc: chucko@jumis.com; Cynthia Shelly; david.bolter@gmail.com; Frank
Olivier; janina@rednote.net; jbrewer@w3.org
Subject: Re: draft of hit testing on active regions (actually paths) in
canvas



Jonas,

We discussed your points on the PF call today. Apple was present. I will
write up a revised draft that associated drawing to the elements per your
request.

Thanks for you feedback.

Rich


Rich Schwerdtfeger
CTO Accessibility Software Group

Inactive hide details for Richard Schwerdtfeger---10/31/2011 11:03:00
AM---Rich Schwerdtfeger CTO Accessibility Software GroupRichard
Schwerdtfeger---10/31/2011 11:03:00 AM---Rich Schwerdtfeger CTO
Accessibility Software Group

From: Richard Schwerdtfeger/Austin/IBM@IBMUS
To: Jonas Sicking <jonas@sicking.cc>,
Cc: chucko@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com,
franko@microsoft.com, janina@rednote.net, jbrewer@w3.org,
public-canvas-api@w3.org, public-html-a11y@w3.org
Date: 10/31/2011 11:03 AM
Subject: Re: draft of hit testing on active regions (actually paths) in
canvas






Rich Schwerdtfeger
CTO Accessibility Software Group

Jonas Sicking <jonas@sicking.cc> wrote on 10/27/2011 07:42:40 PM:

> From: Jonas Sicking <jonas@sicking.cc>
> To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
> Cc: franko@microsoft.com, chucko@jumis.com, david.bolter@gmail.com,
> cyns@exchange.microsoft.com, public-html-a11y@w3.org,
> janina@rednote.net, jbrewer@w3.org, public-canvas-api@w3.org
> Date: 10/27/2011 07:43 PM
> Subject: Re: draft of hit testing on active regions (actually paths)in
canvas
>
> Some feedback:
>
> Limiting to only allowing paths seems like a unfortunate limitation.
> For example it misses calls to drawImage which I think is quite
> common. I'd rather prefer a call where you say "all drawing operations
> from this point should be considered drawing for element X", then let
> the page do arbitrary drawing operations, then have a second call
> which says "i'm now drawing for element Y" or "I'm now drawing for no
> element".
>

I see. So what would you use compute the bounds? Do you use the all the
points to compute a single rectangle for hit testing? ... or do you compute
multiple paths for an element to do hit testing on?


> What is the use case for the zIndex argument? The actual pixel drawing
> operations hasn't had a need for that so far and instead rely on the
> painters algorithm. It seems better to me to have a direct mapping
> between the drawing operations and the accessibility API.
>
zIndex: Performance and authors may have overlapping drawing objects.

> Why return false rather than throw an exception if the element doesn't
> exist? Also what do you mean by "doesn't exist". If the element
> doesn't exist then how could the page have a reference to it and pass
> that reference to the function? Do you mean "element isn't inside the
> canvas"?
>
Throwing an exception makes sense. I meant the element did not exist. I am
thinking that the author may have content outside fallback such as a drop
down menu. We could limit the elements to the fallback content. I would
like your feedback on that.

> In general I think I prefer the API I proposed in [1]. Can you
> describe what problems you were trying to solve with your changes
> compared to that proposal?
>
> [1]
http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html

>
> / Jonas
>
> On Thu, Oct 27, 2011 at 4:56 PM, Richard Schwerdtfeger
> <schwer@us.ibm.com> wrote:
> > Hi Frank,
> >
> > Thank you for the call today. I am glad we are on the same page onthe
three
> > basic methods for hit testing. I took the basic methods that Jonas had
put
> > out earlier on the list and the discussion we had in San Diego and put
> > together an active regions and hit testing section. I also agree with
you
> > this is for when we have multiple interactive elements within canvas. I
was
> > glad to see that customers are starting to come to Microsoft to talk
about
> > building more interactive widgets with canvas - not because I want them
to
> > but more because it shows we are making the right moves to address
canvas
> > accessibility.
> >
> > Jonas, Charles, Frank, David, Cynthia
> >
> > This is a first draft and the section I would like you to look at is
section
> > 16. We can change the names to whatever we want. I have not yet added
spec.
> > information as to how existing functions like scrollElementIntoView
should
> > work on these elements as they should be included in the layout engine
> > positioning per Jonas earlier feedback. I have also not stated how we
would
> > convert the paths to rectangles for the underlying OS platform
accessibility
> > APIs. We could either do this in the canvas spec. or in the
accessibility
> > API mapping guide that Cynthia is leading. We just need to decide.
> > Personally, I prefer the one stop shopping.
> >
> > I am sure Ian will want to tweak or rewrite the algorithm section
processing
> > as he has his own consistent way of doing that but I took my best shot
at
> > it.
> >
> > Please review section 16.
> >
> > (See attached file: clickableregion.html)
> >
> > Since, like many of you I am going to TPAC, I will not have time to
make
> > changes prior to the face to face. Janina please put a link to this
document
> > in your canvas agenda for TPAC.
> >
> > See you all in California.
> >
> >
> > Rich Schwerdtfeger
> >
>