Re: hit testing and retained graphics

James,

Here is the link to Doug's proposal. There is more on his blog referenced
in the post:
http://lists.w3.org/Archives/Public/public-canvas-api/2011AprJun/0117.html

I am willing to work to develop it more with Doug - if the browser
manufacturers agree this is the right approach. Unfortunately, I don't have
the luxury of developing different approaches up to see if Jell-O sticks to
the wall with people. I would like to see if people on the list, especially
browser manufacturers, think this is the right approach.

At this point it needs more development. The benefit of this approach is
that it makes use of the fallback content of canvas to drive the platform
accessibility API as does the rest of canvas with the added benefit of
using SVG drawing paths to supply the bounds of objects. To me there are a
lot of questions to be worked out like what do we do with the existing SVG
keyboard navigation and how would this effect the browsers layout engine
but those are details to be worked out. I just need to know that this is
the preferred direction to head in.

Rich

Rich Schwerdtfeger
CTO Accessibility Software Group



From:	James Robinson <jamesr@google.com>
To:	Richard Schwerdtfeger/Austin/IBM@IBMUS
Cc:	"Tab Atkins Jr." <jackalmage@gmail.com>, Charles McCathieNevile
            <chaals@opera.com>, Charles Pritchard <chuck@jumis.com>, Henri
            Sivonen <hsivonen@iki.fi>, Karl Dubost <karld@opera.com>,
            "public-canvas-api@w3.org" <public-canvas-api@w3.org>,
            "public-html@w3.org" <public-html@w3.org>,
            "public-html-a11y@w3.org" <public-html-a11y@w3.org>,
            public-html-request@w3.org, Doug Schepers <schepers@w3.org>
Date:	07/11/2011 04:22 PM
Subject:	Re: hit testing and retained graphics





On Mon, Jul 11, 2011 at 2:19 PM, Richard Schwerdtfeger <schwer@us.ibm.com>
wrote:


  Rich Schwerdtfeger
  CTO Accessibility Software Group

  public-html-request@w3.org wrote on 07/08/2011 04:16:38 PM:

  > From: "Tab Atkins Jr." <jackalmage@gmail.com>
  > To: Charles Pritchard <chuck@jumis.com>
  > Cc: Richard Schwerdtfeger/Austin/IBM@IBMUS, Doug Schepers
  > <schepers@w3.org>, Charles McCathieNevile <chaals@opera.com>, Henri
  > Sivonen <hsivonen@iki.fi>, Karl Dubost <karld@opera.com>, "public-

  > canvas-api@w3.org" <public-canvas-api@w3.org>, "public-html@w3.org"
  > <public-html@w3.org>, "public-html-a11y@w3.org" <public-html-
  > a11y@w3.org>, public-html-request@w3.org
  > Date: 07/08/2011 04:18 PM

  > Subject: Re: hit testing and retained graphics
  > Sent by: public-html-request@w3.org
  >
  > On Fri, Jul 8, 2011 at 1:38 PM, Charles Pritchard <chuck@jumis.com>
  wrote:
  > > On 7/8/2011 11:35 AM, Tab Atkins Jr. wrote:
  > >> On Fri, Jul 8, 2011 at 9:08 AM, Richard Schwerdtfeger<
  schwer@us.ibm.com>
  > >>  wrote:
  > >>> 2. we are roughly 1 feature away from fixing
  > >>> canvas accessibility.
  > >>
  > >> No we're not.  Not by a long shot.  I've given several reasons why
  not
  > >> in the past.
  > >
  > > Citation please. I haven't seen these -several- reasons that we're
  far away
  > > from canvas a11y.
  > >
  > > A11y itself is a huge field ready for innovation; we're not able to
  come to
  > > those innovations while we're still fighting for the basics, like
  bounding
  > > boxes.
  > >
  > > Canvas a11y has been under discussion since 2009, these issues have
  not
  > > been shifting... they're the same ones as they were back then, for
  the same
  > > set of software and tools there were back then.
  > >
  > > We've addressed most of the issues, Richard has personally spoken
  > > with several AT vendors about it, and I've done the code
  > > and AT testing across Windows, OS X and Mobile Safari.
  > >
  > > These are reality-based actions. I'm fine exploring theoretic
  problems,
  > > I'm happy to do that, but I'd like there to be a distinction and
  recognition
  > > of the work that has been done.
  >
  > For one simple example I've brought up several times, and Jonas has
  > brought up in this thread as well, rich text editing. Trying to do
  > this in Canvas is a horrifying trainwreck of problems, both large and
  > larger, of which dealing with bounding boxes is only one.
  >

  The subject is hit testing and retained graphics. We need this regardless
  of the rich text editing discussion. We are losing focus on the problem.

  I think we spend enough cycles on the rich text editing discussion.


  > It appears that you have to solve all of these problems to solve the
  > "use canvas to do remote access" use-case, though, which is a problem.
  >
  >
  > >>> The smoothing of shapes has nothing to do with retained mode.
  > >>
  > >> Um, it has everything to do with retained mode if you want it to be
  > >> automatic.  By definition.
  > >
  > > Again, automatic is a strange term that keeps coming up.
  > > In css you can set something to 100%, which makes it "automatic",
  > > whereas with canvas you do need to monitor for resize events
  > > and gather various metrics to have the same result.
  > >
  > > But gosh that's really common for canvas. It's a low level API.
  >
  > Yes. My point and Doug's is that you have to do it manually.
  > Retained-mode APIs can do it automatically .
  >
  >
  > >>> What we are
  > >>> trying to do is make canvas accessible. SVG is another technology
  and
  > >>> that
  > >>> is a separate accessibility effort.
  > >>
  > >> Conway's Law strikes again.
  > >
  > > This is a subtle put down; I get it, but it's absurd.
  > >
  > > You're telling us that, because our teams are using canvas, and have
  come to
  > > us with issues regarding canvas, that we've become deluded.
  >
  > I wouldn't say "deluded", but I would say that you're focused in an
  > unproductive way.
  >
  > Like I've said before (and Jonas has stated in different terms), the
  > problem is not "how do we make canvas accessible?". It's "how do we
  > solve problem X for users in a way that's accessible?". The correct
  > solution for "how do I do rich text editing in an accessible way" is
  > "use contenteditable", for example. If contenteditable isn't subpar,
  > we can fix those problems more easily than we can add sufficient
  > primitives to allow authors to reinvent contenteditable in canvas.
  >
  We are getting off track with contenteditable. This discussion is simply
  about providing the AT the location and bounds of an object.

  >
  > > As it happens, Canvas is a low level API, and SVG a high level
  format.
  > > We're trying to fix the low level API before taking on the high level
  > > format.
  >
  > SVG is made out of rectangles, circles, and paths. You stroke them
  > and fill them. It's roughly the same level as canvas. Their only
  > difference is the constantly-mentioned retained vs immediate
  > distinction.
  >
  > (SVG does include some higher-level constructs like filters, but
  > that's irrelevant to the discussion at hand.)
  >
  >
  OK. ... What do you think of Doug Schepers proposal to have an API that
  would integrate SVG into Canvas such that the retained mode aspects are
  limited to SVG?


  > >>> If SVG is so wonderful why are you not
  > >>> pushing for an SVG element in HTML 5?
  > >>
  > >> SVG is in HTML.<http://whatwg.org/html#svg-0>
  > >
  > > More appropriate link:
  > > http://www.w3.org/TR/html5/the-map-element.html#svg-0
  > >
  > > SVG is mentioned in HTML. Processing is not specified in that
  document.
  > > Object tags are mentioned in HTML as well, that doesn't mean that
  Java
  > > is part of HTML5.
  >
  > I was flippantly answering what I thought was a rude and unproductive
  > question. Don't worry too much over the details. ^_^
  >
  >
  > > Regardless, I think SVG is wonderful, and I'd rather be spending my
  > > time addressing issues on SVG. But I'm stuck here, in the low level
  > > API, until the foundation is solid.
  > >
  > > There's a very near and dear issue to me with SVG, and that's
  > > the capability to display InkML rendering. SVG is currently quite
  poor
  > > at handling pressure values on lines, and using repeated elements
  > > really takes a toll on most implementations.
  >
  > You mean like a single line that gets thinner and thicker at different
  > points? SVG handles that currently by making the entire stroke a path
  > and filling it. It would be easier if you could provide a variable
  > stroke width instead, though.
  >
  >
  > >>> Why have you not pushed to ditch
  > >>> canvas from HTML in favor of SVG?
  > >>
  > >> Because that would be a stupid thing to push for, frankly.  Doug's a
  > >> smart guy.  He recognizes, like everyone else, that immediate mode
  > >> APIs have an advantage over retained mode APIs in some
  circumstances.
  > >
  > > There's WebGL, a mix of immediate and retained mode API; keep that in
  mind.
  > > It works because it should. It's closer a format for efficient InkML
  > > processing
  > > than either Canvas or SVG.
  > >
  > > Again, we're looking at low level API and high level format;
  > > this confusion over retained mode is a red herring.
  >
  > No, it's not. Several of the problems you've presented are solved
  > *much* more easily if you start with a retained-mode graphics format.
  > That's one of the major points we keep trying to make - you've put
  > blinders on and limited yourself to solving problems using *only* the
  > 2d canvas context. I guarantee that this will make your solutions
  > much worse than if you try and solve them with the best tool for each
  > job.
  >
  The problem is authors don't always agree with what we might think is the
  best tool for the job. I would really like to hear you weigh in on Doug
  Schepers proposal. We can't leave canvas inaccessible. Authors will use
  canvas to draw objects on the screen and some will want what they produce
  to be accessible. We need to provide them the tools to support
  interoperability with assistive technologies. They need to provide the
  bounds of an object to meet that requirement.

  Would what Doug is suggesting, using SVG as Doug suggested be an
  acceptable solution to Webkit and Chrome?

That's impossible to evaluate without a concrete proposal.

- James



  > ~TJ
  >