Canvas and access to remote desktop environments [Was: HTML 5 Canvas Accessibility Call on January 10]

On Sun, Jan 16, 2011 at 4:35 PM, Benjamin Hawkes-Lewis
<bhawkeslewis@googlemail.com> wrote:
> On Fri, Jan 14, 2011 at 5:37 PM, Richard Schwerdtfeger
> <schwer@us.ibm.com> wrote:
>> I think the IE team needs to look at the following:
>>
>> GTK client on HTML5 canvas:
>> http://blogs.gnome.org/alexl/2010/11/23/gtk3-vs-html5/
>>
>> noVNC - VNC client using HTML5 Canvas:
>> http://kanaka.github.com/noVNC/

[snip]

> I think canvas does have a potential role to play in addressing the
> remote desktop environment use case, but I don't think the proposed
> features help. I'll discuss that use case in a new thread.

As far as I know, VNC and similar protocols tunnel pixel and sound in
and keyboard presses, mouse events, and sound out. They do not tunnel in
the information necessary to drive remote assistive technology or tunnel
out the commands necessary to drive remote environments
programmatically.

Today's solutions to the problem seem to include:

1. Firing up a dedicated server on the remote system to tunnel
accessibility traffic back and forth directly between the remote system
and the local assistive technology. This is how JAWS provides remote
access.

2. Firing up a remote instance of the assistive technology and pulling
in sound and braille events into devices on the local system rather than
trying to interact with the remote system via local AT. This is how
Window-Eyes provides remote access to Windows systems.

Generalizing the first model to the web is problematic. I can imagine
producing a standardized accessibility API with mappings to desktop
APIs. But Windows accessibility depends on a huge series of hacks that
depend on specialized interpretations of the accessibility tree, video
intercepts, specialized APIs for Office and the DOM, and so on.
Consequently, simply converting the UI Automation accessibility tree to
a standard format and then pushing it over the network would be fairly
useless to client applications. Expecting developers of remote access
servers to map not only the accessibility tree, but also all the other
APIs that Windows accessibility applications depend on, into a
standardized accessibility tree seems unrealistic. We could forget about
supporting todays' Windows applications and move forward on the basis
that desktop applications should make use of system widget sets, and
when they make custom widgets use system accessibility features, rather
than forcing AT to depend on hacks. But even then we're talking about a
huge, politically complicated, technically tangled project. RTE support
is a tiny part of the puzzle. On the other hand, Freedom Scientific's
Remote Access software probably could work just as well alongside
visuals served by a webapp VNC client as it does beside a desktop VNC
client, and could depend on the remote server for selection and caret
information. RTE features for HTML5 canvas don't really help here.

In the short term, I think it would be more realistic to webify the
second model. HTML has <canvas> for bitmapped visual output, <audio> for
sound output, DOM event support for keyboard and mouse events. There has
been discussion around adding features for visual and audio input (in
particular for webcams) and other devices such as bluetooth. If we added
support for braille output (a <braille> element maybe?), then webapp
clients could register themselves as braille output devices for remote
systems via remote access servers. Again, RTE support for canvas doesn't
really help here.

Thoughts?

--
Benjamin Hawkes-Lewis

Received on Sunday, 16 January 2011 16:42:37 UTC