Re: 3D Pointers

Hi Jacob and Art and others,

Thank you for following up on this, and for adding a reference to this
thread within the notes on pointer event extensibility (
http://www.w3.org/wiki/PointerEvents/UseCasesAndRequirements).

The EdgeConf discussion (
https://www.youtube.com/watch?v=zxuA-wyajhY&feature=player_detailpage#t=2454s)
provides excellent context for this thread, but I find that I very
respectfully disagree when Boris Smus argues against the introduction of
the Z-coordinate, stating, "It's unclear what the units would be for all
this stuff.  You're breaking the connection ... of mapping to a screen
coordinate.  As soon as you're dealing with tracking real world stuff, it's
in some different coordinate system.  If you can bring it back to screen
space, you're doing it with some weird transform."

I think a transformed unit is actually an ideal way of expressing the Z
dimension on the Web, and its precedent exists in CSS transforms.  If we
have a CSS perspective, then the browser should be able to map from the
device driver data to the web developer's intended use of 3D space.  That
is, all we really need is a *relative* unit to come from the device
drivers, and the established perspective would provide a mapping into
Z-axis pixels to dovetail with CSS transforms.  For example, if the spec
asked for a floating point value from 1 to -1 to come from the device
driver data, the browser could easily transform this data into something
easy for the web developer to use, based on how the developer has set up
the 3D space.  I'm no expert on WebGL, but I did discuss this with a WebGL
developer, and I believe this relative/transformed approach would work
there as well.

The underlying issue here is probably an old debate at the W3C -- the
question of whether the Web Platform should get out in front of emerging
technology so as to compete effectively with native and proprietary
platforms, or whether it is better to first "allow a thousand flowers to
bloom" and wait to see what patterns emerge from the marketplace.  My own
opinion is that for the Web to become the platform that we all want it to
be, it needs to give both browsers and device manufacturers something to
aim toward.  Otherwise we have a thousand Betamax flowers, no VHS, and the
Web is left lagging behind with Super8.

I think this discussion resonates, to a small degree, with the discussion
on the "Last Call comments" thread about tiltX/Y vs. spherical coordinates.
 That is, the spec appears to be aimed almost exclusively at mouse, touch
and stylus input, and I would very much like to see the spec broaden its
vision and scope.

One change to the spec could achieve this: the introduction of a
Z-coordinate.  If this would require too much processor power, I could see
that as a reason to not implement this.  Otherwise, I think the spec
shortchanges the Web by not including this kind of data.  If we have to
wait for v2 to see this data, 3D motion detection on the Web will be
totally fragmented until that happens, partially and significantly
crippling the Web's ability to take advantage of this new input paradigm.

- Bill


On Mon, Mar 18, 2013 at 12:07 PM, Jacob Rossi <Jacob.Rossi@microsoft.com>wrote:

> On Mon, Mar 18, 2013 at 4:09 AM, Arthur Barstow <art.barstow@nokia.com>
> wrote:
> >
> > Hi Bill,
> >
> > FYI, Rick Byers started a related thread "pointerType extensibility"
> last February <
> http://lists.w3.org/Archives/Public/public-pointer-events/2013JanMar/0096.html>.
> During our March 12 call we discussed the extensibility issue again and
> agreed to:  keep the existing functionality for v1 <
> http://www.w3.org/2013/03/12-pointerevents-minutes.html#item03>; add the
> extensibility issue to the v2 feature list <
> http://www.w3.org/wiki/PointerEvents/UseCasesAndRequirements>.
> >
> > -ArtB
> >
> >
> > On 3/15/13 7:22 PM, ext Bill Fisher wrote:
> >>
> >> Hi,
> >>
> >> I'm wondering if there are any plans to incorporate 3D pointers coming
> from a motion capture device such as Leap Motion. (video:
> http://www.youtube.com/watch?v=_d6KuiuteIA , home: https://leapmotion.com)
> >>
> >> I do not see any such capabilities in the current proposed
> specification, and this would appear to be a great limitation of the
> proposed pointer events as we move toward post-touch interfaces based on
> similar controllers.
> >>
> >> Leap Motion, for example, has their own API to represent "pointables"
> in 3D space, and this is made available to web pages through Web Sockets
> and Node.js via the controller software.
> >>
> >> But it seems to me like it would be a better idea to create a
> specification where this information could be exposed to the browser
> directly.
> >>
> >> For more info on how Leap Motion is representing 3D pointables in
> JavaScript, please see:
> >> https://github.com/leapmotion/leapjs
> >> http://leapmotion.github.com/leapjs/index.html
> >> And perhaps most notably:
> http://leapmotion.github.com/leapjs/Pointable.html
>
> As Art mentioned above,  we've had the related discussion about adding new
> device types. The model is extensible by new device types and we'll be
> surely be discussing this further in a future version of the spec.
>
> On the specific question of 3D spatial input, I'd encourage you to watch a
> short commentary from Edge Conf this year on this subject. [1] FWIW, I
> think several of us (myself included) have preordered a Leap Motion for
> experimentation with respect to Pointer Events. But I think the comments by
> the panelists at Edge Conf resonate with me very well. In particular,
>
> 1) The device landscape is pretty limited at the moment. Jumping to design
> an API tailored to just one or two retail devices will likely end up not
> being what we want in the long run and ultimately slow us down.
> 2) We don't yet have a full picture of the use cases for web sites yet. If
> we wait a bit for the scenarios to bloom, then we can design an API that's
> better for the long run.
> 3) In line with #1 and #2, it's not clear what the right units for the web
> should be. The x and y coordinates map to screen coordinates. But depth (z)
> isn't a screen coordinate.  In LeapJS, for example, the units are
> millimeters for x, y, and z. For most scenarios we're targeting with
> Pointer Events, millimeters wouldn't be appropriate. Then also consider,
> how would we rationalize the design for other motion sensing devices like
> Xbox Kinect (which deals in a larger space) or the Samsung Galaxy S4's "Air
> View" (which deals in a closer space)?
>
> While I think applying motion tracking devices to Pointer Events is an
> excellent idea, I think the ecosystem, devices, and use case development
> aren't yet mature enough to codify an API design into a web standard. So I
> think our previous decision to tackle extending PE to support additional
> device types in the V2 spec is still correct here.
>
> >> If I'm not mistaken, I think this email to the list might constitute
> "raising an issue".  But I am not sure about the correct protocol to do
> that.  If anyone would care to let me know, please do.
>
> Yes, this mail is all that's required to raise an issue. Thanks for
> contributing!  We can continue this discussion over mail and possibly
> during our next schedule teleconference (likely  26 March at 8am PST). [2]
>
> -Jacob
>
> [1]
> https://www.youtube.com/watch?v=zxuA-wyajhY&feature=player_detailpage#t=2454s(time: 40:45 - 46:00)
> [2] http://www.w3.org/wiki/PointerEvents/Meetings
>
>


-- 
Bill Fisher
510.414.8771
http://fisherwebdev.com