Exploring HiFi Events

Continuing on Jacob Rossi's comments about clientZ and 3D input devices 
such as the Kinect,
and my comments on simple pen tablets...
http://lists.w3.org/Archives/Public/www-dom/2011JulSep/0125.html

I'm still holding out hope that we can smuggle "force" or "pressure" 
into DOM events, as pen pressure is a simple floating point, and 
pressure devices have been around for ages. It's a shame that they've 
never made it into the mainstream events standard.

That's an issue I'll continue to work on...

Aside form that issue, and more to the point of this thread:
HD mice and pens send a good 10x more events than typical mouse 
interaction. They are high precision.

This leads to something else: with many applications, mouse events are 
the primary entry point -- that is, as the user moves their mouse, the 
app receives events, and repaints the screen based on those events. If 
the system is slow, it simply processes less events (or with less 
fidelity).  When working with more high density data, it becomes 
counter-productive to repaint so frequently. And so the repaints should 
really be queued for each requestAnimationFrame / setTimeout cycle, 
instead of being executed on each input event. That of course, takes 
extra effort for the programmer, and it's something that could be 
improved in the UA, by the use of buffers.

The postMessage structured clone algorithm for ArrayBuffer, and the 
reserved names of InkML could work very well for introducing a new, HiFi 
event class. Events would only be sent every ~30ms (raf), and they would 
be sent in an ArrayBuffer using postMessage semantics. It would work as 
well for pen/hd mouse events as it would for sending a raw Kinect point 
cloud for use in WebGL.

Transfer of ArrayBuffer ownership was added to the HTML5 spec recently 
(Transferable types):
http://lists.w3.org/Archives/Public/public-html-diffs/2011Jun/0130.html

InkML traceFormat is a useful guide for cases.
http://www.w3.org/TR/InkML/

Lacking an actual interface to use, it's likely that in introductions or 
plugins, I may just re-use postMessage, perhaps abusing the origin field 
to describe the source.

This is, in pseudocode, what I envision:
postMessage( [ {'Z': [type: 'decimal', units: 'cm']} ], 
'event://myInputSource', [ ArrayBuffer(....) ]);

It'd buffer-up events, and post a message every ~30ms, no quicker than 
RAF would run, and allowably, it could run slower.

No data is lost, and it'd work as well for mouse events as a large 
connect point cloud. Further, both could be sent directly to WebGL, 
which is nice.


Well, that's my exploration of HiFi events for now. I hope to see some 
discussion on this, as it'd be wonderful to have a means to incorporate 
these new data streaming types.

-Charles

Received on Wednesday, 17 August 2011 19:34:31 UTC