Re: draft of hit testing on active regions (actually paths) in canvas

What happens if beginElement is called twice? I.e. for code like:

beginElement(elem1);
... drawing operations here ...
beginElement(elem2);
... drawing operations here ...
endElement(elem2);
... drawing operations here ...
endElement(elem1);

And what happens if the endElement call doesn't match the previous
beginElement call? I.e. code like:

beginElement(elem1);
... drawing operations here ...
endElement(elem2);

It seems like having a single functions, rather than a pair, avoids a lot
of these types of edge cases.

Additionally, I'd really like to hear the input from Charles Pritchard on
why a setPath-style API fits better with existing code. If that's indeed
the case then we might be able to much quicker get deployment of this API
on websites.

/ Jonas

On Mon, Oct 31, 2011 at 4:46 PM, Frank Olivier
<Frank.Olivier@microsoft.com>wrote:

>  A preview of that draft, based on notes I took during the meeting:****
>
> ** **
>
> These new methods manage the association of pixels drawn with canvas
> fallback elements:****
>
> beginElement(Element element);****
>
> endElement(Element element);****
>
> eraseElement(element) ;****
>
> eraseElements();****
>
> ** **
>
> So, code example:****
>
> ** **
>
> For a html page with this content:****
>
> <canvas>****
>
>   <input type=checkbox id=checkboxA >Checkbox A</input>****
>
>   <input type=checkbox id=checkboxB >Checkbox B</input>****
>
> </canvas>****
>
> ** **
>
> By default, all pixels on a canvas are ‘unassociated’. The author would
> use these new methods to associate pixels with fallback elements.****
>
> ** **
>
> When drawing the checkbox:****
>
> // Rendering code****
>
> ctx.beginElement(checkboxA);     //Tell the canvas context that all
> non-fully-transparent pixels that we are rendering from this point on
> represents the clickable area for the checkbox UI****
>
> // Draw a background for the checkbox****
>
> ctx.fillStyle = 'white’;****
>
> ctx.rect(0, 0, 100, 20);****
>
> context.fill();****
>
> ** **
>
> // Draw the actual checkbox****
>
> ctx.rect(5, 5, 10, 10);****
>
> ctx.stroke();****
>
> ** **
>
> if (checkboxA.checked) ****
>
> {****
>
>   ctx.fillStyle = 'black';****
>
>   ctx.fill();****
>
> }****
>
> ctx.fillText(‘checkbox’, 20, 10);****
>
> endElement(checkboxA); //We are done with element rendering****
>
> ** **
>
> The UA/magnifier would associate any pixels rendered from beginElement()
> to endElement() with the fallback dom element; the fallback dom element
> ‘owns’ those pixels on the screen. To resolve overlap, last write wins.
> (Implementation note: A UA could use a bitmap to associate pixels with
> fallback dom elements. A value of 0 in that bitmap could mean ‘no
> association’; value 1 could mean ‘first element in the fallback dom element
> lookup table’, etc.)****
>
> ** **
>
> Between beginElement() and endElement(), the UA would keep track of the
> extent of the pixels that are touched by drawing operations; it would use
> this to compute the bounding rectangle when a magnifier asks for the
> element’s extent.****
>
> ** **
>
> At some point in the future, when my canvas bitmap no longer has that
> checkbox visible:****
>
> ** **
>
>                ctx.eraseElement(checkboxA); //Clear all pixels ‘owned’ by
> checkboxA****
>
>                (This imo assumes that you are removing or disabling the
> corresponding DOM checkbox input element)****
>
> ** **
>
> All the pixels that were associated with checkboxA would now be
> unassociated with any fallback dom element.****
>
> ** **
>
> eraseElements() does the same, but for all fallback dom elements.****
>
> ** **
>
> I could (potentially) also clear some of the pixels that are associated
> with a corresponding canvas element:****
>
> ctx.beginElement(null);   //Tell the canvas context that all
> non-fully-transparent pixels that we are rendering from this point on
> should not be part of any element****
>
> ctx.fillStyle = 'white’;****
>
> ctx.rect(0, 0, 100, 100);****
>
> context.fill();****
>
> endElement(checkboxA); //We are done with element rendering****
>
>              ****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Richard Schwerdtfeger [mailto:schwer@us.ibm.com]
> *Sent:* Monday, October 31, 2011 4:32 PM
> *To:* Jonas Sicking; public-canvas-api@w3.org; public-html-a11y@w3.org
> *Cc:* chucko@jumis.com; Cynthia Shelly; david.bolter@gmail.com; Frank
> Olivier; janina@rednote.net; jbrewer@w3.org
>
> *Subject:* Re: draft of hit testing on active regions (actually paths) in
> canvas****
>
>  ** **
>
> Jonas,
>
> We discussed your points on the PF call today. Apple was present. I will
> write up a revised draft that associated drawing to the elements per your
> request.
>
> Thanks for you feedback.
>
> Rich
>
>
> Rich Schwerdtfeger
> CTO Accessibility Software Group
>
> [image: Inactive hide details for Richard Schwerdtfeger---10/31/2011
> 11:03:00 AM---Rich Schwerdtfeger CTO Accessibility Software Group]Richard
> Schwerdtfeger---10/31/2011 11:03:00 AM---Rich Schwerdtfeger CTO
> Accessibility Software Group
>
> From: Richard Schwerdtfeger/Austin/IBM@IBMUS
> To: Jonas Sicking <jonas@sicking.cc>,
> Cc: chucko@jumis.com, cyns@exchange.microsoft.com, david.bolter@gmail.com,
> franko@microsoft.com, janina@rednote.net, jbrewer@w3.org,
> public-canvas-api@w3.org, public-html-a11y@w3.org
> Date: 10/31/2011 11:03 AM
> Subject: Re: draft of hit testing on active regions (actually paths) in
> canvas****
>  ------------------------------
>
>
>
>
>
>
> Rich Schwerdtfeger
> CTO Accessibility Software Group
>
> Jonas Sicking <jonas@sicking.cc> wrote on 10/27/2011 07:42:40 PM:
>
> > From: Jonas Sicking <jonas@sicking.cc>
> > To: Richard Schwerdtfeger/Austin/IBM@IBMUS,
> > Cc: franko@microsoft.com, chucko@jumis.com, david.bolter@gmail.com,
> > cyns@exchange.microsoft.com, public-html-a11y@w3.org,
> > janina@rednote.net, jbrewer@w3.org, public-canvas-api@w3.org
> > Date: 10/27/2011 07:43 PM
> > Subject: Re: draft of hit testing on active regions (actually paths)in
> canvas
> >
> > Some feedback:
> >
> > Limiting to only allowing paths seems like a unfortunate limitation.
> > For example it misses calls to drawImage which I think is quite
> > common. I'd rather prefer a call where you say "all drawing operations
> > from this point should be considered drawing for element X", then let
> > the page do arbitrary drawing operations, then have a second call
> > which says "i'm now drawing for element Y" or "I'm now drawing for no
> > element".
> >
>
> I see. So what would you use compute the bounds? Do you use the all the
> points to compute a single rectangle for hit testing? ... or do you compute
> multiple paths for an element to do hit testing on?
>
>
> > What is the use case for the zIndex argument? The actual pixel drawing
> > operations hasn't had a need for that so far and instead rely on the
> > painters algorithm. It seems better to me to have a direct mapping
> > between the drawing operations and the accessibility API.
> >
> zIndex: Performance and authors may have overlapping drawing objects.
>
> > Why return false rather than throw an exception if the element doesn't
> > exist? Also what do you mean by "doesn't exist". If the element
> > doesn't exist then how could the page have a reference to it and pass
> > that reference to the function? Do you mean "element isn't inside the
> > canvas"?
> >
> Throwing an exception makes sense. I meant the element did not exist. I am
> thinking that the author may have content outside fallback such as a drop
> down menu. We could limit the elements to the fallback content. I would
> like your feedback on that.
>
> > In general I think I prefer the API I proposed in [1]. Can you
> > describe what problems you were trying to solve with your changes
> > compared to that proposal?
> >
> > [1]
> http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html
> >
> > / Jonas
> >
> > On Thu, Oct 27, 2011 at 4:56 PM, Richard Schwerdtfeger
> > <schwer@us.ibm.com> wrote:
> > > Hi Frank,
> > >
> > > Thank you for the call today. I am glad we are on the same page onthe
> three
> > > basic methods for hit testing. I took the basic methods that Jonas had
> put
> > > out earlier on the list and the discussion we had in San Diego and put
> > > together an active regions and hit testing section. I also agree with
> you
> > > this is for when we have multiple interactive elements within canvas.
> I was
> > > glad to see that customers are starting to come to Microsoft to talk
> about
> > > building more interactive widgets with canvas - not because I want
> them to
> > > but more because it shows we are making the right moves to address
> canvas
> > > accessibility.
> > >
> > > Jonas, Charles, Frank, David, Cynthia
> > >
> > > This is a first draft and the section I would like you to look at is
> section
> > > 16. We can change the names to whatever we want. I have not yet added
> spec.
> > > information as to how existing functions like scrollElementIntoView
> should
> > > work on these elements as they should be included in the layout engine
> > > positioning per Jonas earlier feedback. I have also not stated how we
> would
> > > convert the paths to rectangles for the underlying OS platform
> accessibility
> > > APIs. We could either do this in the canvas spec. or in the
> accessibility
> > > API mapping guide that Cynthia is leading. We just need to decide.
> > > Personally, I prefer the one stop shopping.
> > >
> > > I am sure Ian will want to tweak or rewrite the algorithm section
> processing
> > > as he has his own consistent way of doing that but I took my best shot
> at
> > > it.
> > >
> > > Please review section 16.
> > >
> > > (See attached file: clickableregion.html)
> > >
> > > Since, like many of you I am going to TPAC, I will not have time to
> make
> > > changes prior to the face to face. Janina please put a link to this
> document
> > > in your canvas agenda for TPAC.
> > >
> > > See you all in California.
> > >
> > >
> > > Rich Schwerdtfeger
> > >
> > ****
>