Re: hit testing and retained graphics

On 7/8/2011 2:16 PM, Tab Atkins Jr. wrote:
> On Fri, Jul 8, 2011 at 1:38 PM, Charles Pritchard<chuck@jumis.com>  wrote:
>> On 7/8/2011 11:35 AM, Tab Atkins Jr. wrote:
>>> On Fri, Jul 8, 2011 at 9:08 AM, Richard Schwerdtfeger<schwer@us.ibm.com>
>>>   wrote:
>>>> 2. we are roughly 1 feature away from fixing
>>>> canvas accessibility.
>>> No we're not.  Not by a long shot.  I've given several reasons why not
>>> in the past.
>> Citation please. I haven't seen these -several- reasons that we're far away
>> from canvas a11y.
> For one simple example I've brought up several times, and Jonas has
> brought up in this thread as well, rich text editing.  Trying to do
> this in Canvas is a horrifying trainwreck of problems, both large and
> larger, of which dealing with bounding boxes is only one.


These issues are more related to DOM than Canvas. Hironori Bono has 
brought up
IME extensions as well as spell check extensions. Though they did not 
see acceptance
in the WHATWG mailing list, he has continued to push forward on them.

I've not found RTE to be a horrifying train-wreck. We followed the CSS 
line box specs
and things worked out.

I'm aware of the neat issues surrounding this kind of work. Regardless, 
RTE was
ruled out as a use case by all major vendors last year. I'd imagine that 
future
developments on contentEditable will enable Canvas RTE as they add new 
vocabulary.

> It appears that you have to solve all of these problems to solve the
> "use canvas to do remote access" use-case, though, which is a problem.


We can only improve remote access, not "solve it".

Web Apps has a wealth of functionality to help us with remote access,
from binary data options to dataTransfer semantics.

VNC clients have proven that a web app can run as well as a desktop app 
for remote access.

We now have caret location, we have text selection (via DOM range), 
binary transfer
and drag/drop support and focus events. Full screen mouse capture is 
under discussion,
headway is being made.

That's pretty good.


>>>> The smoothing of shapes has nothing to do with retained mode.
>>> Um, it has everything to do with retained mode if you want it to be
>>> automatic.  By definition.
>> Again, automatic is a strange term that keeps coming up.
>> In css you can set something to 100%, which makes it "automatic",
>> whereas with canvas you do need to monitor for resize events
>> and gather various metrics to have the same result.
>>
>> But gosh that's really common for canvas. It's a low level API.
> Yes.  My point and Doug's is that you have to do it manually.
> Retained-mode APIs can do it automatically .


Yes, in Canvas you have to do things like resolution management
and repainting in the scripting environment. It's true.

With the canvas API must create your own scene graph to meet your 
project needs.
You're talking about a few lines of code, to manage these "automatic" needs.

Yes, Canvas is more script heavy than SVG.

Historically (and currently) scenes can be rendered faster in Canvas
than in SVG implementations on the mobile and desktop. Currently,
there is wider support for Canvas-based SVG rendering than UA based
rendering.

When it comes to it, I start with Canvas and target SVG second,
for purposes of compatibility and speed. As for the costs of implementation,
it really depends on the application.

SVG FE and VE are not widely supported, which has until WebGL, knocked
out SVG as an option for some applications.

SVG scenes require a rigid DOM structure -- so generally, I will use it
as an output/input/interop format, but I don't work on a live SVG tree,
as it's much easier to work on strings, arrays and objects in JS than a 
DOM tree.

SVG is sorely lacking a replicate tag. With a few tweaks to the spec,
SVG could have a retained bitmap option, which would help me around most
of the issues I've encountered. There is an accumulate option for 
similar performance reasons.

> Like I've said before (and Jonas has stated in different terms), the
> problem is not "how do we make canvas accessible?".  It's "how do we
> solve problem X for users in a way that's accessible?".  The correct
> solution for "how do I do rich text editing in an accessible way" is
> "use contenteditable", for example.  If contenteditable isn't subpar,
> we can fix those problems more easily than we can add sufficient
> primitives to allow authors to reinvent contenteditable in canvas.
Jonas said, how do we make application X accessible.
I believe that's what he said anyway.

The list has seen many application X examples.

I'm pretty sure you and Jonas are saying two different things.


>> As it happens, Canvas is a low level API, and SVG a high level format.
>> We're trying to fix the low level API before taking on the high level
>> format.
> SVG is made out of rectangles, circles, and paths.  You stroke them
> and fill them.  It's roughly the same level as canvas.  Their only
> difference is the constantly-mentioned retained vs immediate
> distinction.
It's made of complex use tags and transforms, of references and of 
animations,
and of css inheritance, of caching and resolution management and 
additional stroke styles.

Most of those items are handled by the author in canvas; their 
management is handled by the implementation
in SVG. It's a higher level language.

>> Regardless, I think SVG is wonderful, and I'd rather be spending my
>> time addressing issues on SVG. But I'm stuck here, in the low level
>> API, until the foundation is solid.
>>
>> There's a very near and dear issue to me with SVG, and that's
>> the capability to display InkML rendering. SVG is currently quite poor
>> at handling pressure values on lines, and using repeated elements
>> really takes a toll on most implementations.
> You mean like a single line that gets thinner and thicker at different
> points?  SVG handles that currently by making the entire stroke a path
> and filling it.  It would be easier if you could provide a variable
> stroke width instead, though.
We're not able to do nice tapering of edges, and the calculations 
required to make the path,
as well as to write it out (you have to do a loop, essentially) really 
are a lot of extra work.

We lose directional information, which is a shame. I've tried a few 
tests to see how I can
replicate things, when it comes to it, I can use InkML and walk the DOM 
with the scripting
environment, but I end up with thousands of DOM nodes to get the desired 
effect.

Render time is 10x as slow on Mobile Safari with the SVG scene as with 
the Canvas scene,
further, any zooming in triggers repainting mechanisms which can't be 
managed by the author.

With Canvas, I can manage repainting and bitmap resolution, which is 
really handy
for responsiveness.



>>>> Why have you not pushed to ditch
>>>> canvas from HTML in favor of SVG?
>>> Because that would be a stupid thing to push for, frankly.  Doug's a
>>> smart guy.  He recognizes, like everyone else, that immediate mode
>>> APIs have an advantage over retained mode APIs in some circumstances.
>> There's WebGL, a mix of immediate and retained mode API; keep that in mind.
>> It works because it should. It's closer a format for efficient InkML
>> processing
>> than either Canvas or SVG.
>>
>> Again, we're looking at low level API and high level format;
>> this confusion over retained mode is a red herring.
> No, it's not.  Several of the problems you've presented are solved
> *much* more easily if you start with a retained-mode graphics format.
> That's one of the major points we keep trying to make - you've put
> blinders on and limited yourself to solving problems using *only* the
> 2d canvas context.  I guarantee that this will make your solutions
> much worse than if you try and solve them with the best tool for each
> job.

But I've shown you the code that's necessary to solve the problems I've 
mentioned.
It takes two additional method calls to bind text to an area on the 
screen. That's not difficult.

I've also looked at the code bases for Mozilla and WebKit -- the very 
basics that I need
to solve the issue (sending information to the Accessibility Tree) take 
less than a dozen
lines of code in each branch.

The more complex items brought up would take more code, but they're
certainly no -large- undertaking.

It can't be *much* more easily when the solution at hand is not that 
difficult.

If you want more code samples, to help you understand the proposals,
I'm willing to do that. I'm willing to point at line numbers in the 
WebKit and Mozilla
code bases as well, and demonstrate which function calls are necessary 
to pass
information to the accessibility tree.

I mean, I can work with *easy* as a qualifier. It simply has not been a 
qualifier here,
*correctness* has been the qualifier. And by all measure, the *correct* 
means, in the
opinion of many vendors, is to use SVG and simply -not- use Canvas.

-Charles

Received on Friday, 8 July 2011 22:35:44 UTC