W3C home > Mailing lists > Public > public-webapps@w3.org > January to March 2014

Re: Proposal Virtual Reality "View Lock" Spec

From: Rob Manson <robman@mob-labs.com>
Date: Thu, 27 Mar 2014 06:25:29 +1100
Message-ID: <533329A9.9090807@mob-labs.com>
To: public-webapps@w3.org
Hi,

we've already implemented an open source library that integrates WebGL, 
WebAudio, DeviceOrientation and gUM to create an easy to use VR/AR 
framework that runs on Rift, Glass, mobile, tablet, pc, etc.

Here's an overview of our API.

   https://buildar.com/awe/tutorials/intro_to_awe.js/index.html

And the project is in our github repos.

   https://github.com/buildar/awe.js

Hope that's relevant.

PS: We're also working on a Depth Stream Extension proposal to add depth 
camera support to gUM.

roBman


On 27/03/14 5:58 AM, Lars Knudsen wrote:
> I think it could make sense to put stuff like this as an extension on 
> top of WebGL and WebAudio as they are the only two current APIs close 
> enough to the bare metal/low latency/high performance to get a decent 
> experience.  Also - I seem to remember that some earlier generation VR 
> glasses solved the game support problem by providing their own GL and 
> Joystick drivers (today - probably device orientation events) so many 
> games didn't have to bother (too much) with the integration.
>
> In theory - we could:
>
>  - extend (if needed at all) WebGL to provide stereo vision
>  - hook up WebAudio as is (as it supports audio objects, Doppler 
> effect, etc. similar to OpenAL
>  - hook up DeviceOrientation/Motion in Desktop browsers if a WiiMote, 
> HMD or other is connected
>  - hook up getUserMedia as is to the potential VR camera
>
> ..and make it possible to do low latency paths/hooks between them if 
> needed.
>
> It seems that all (or at least most) of the components are already 
> present - but proper hooks need to be made for desktop browsers at 
> least (afaik .. it's been a while ;))
>
> - Lars
>
>
> On Wed, Mar 26, 2014 at 7:18 PM, Brandon Jones <bajones@google.com 
> <mailto:bajones@google.com>> wrote:
>
>     So there's a few things to consider regarding this. For one, I
>     think your ViewEvent structure would need to look more like this:
>
>     interface ViewEvent : UIEvent {
>     readonly attribute Quaternion orientation; // Where Quaternion is
>     4 floats. Prevents gimble lock.
>     readonly attribute float offsetX; // offset X from the calibrated
>     center 0 in millimeters
>     readonly attribute float offsetY; // offset Y from the calibrated
>     center 0 in millimeters
>     readonly attribute float offsetZ; // offset Z from the calibrated
>     center 0 in millimeters
>         readonly attribute float accelerationX; // Acceleration along
>     X axis in m/s^2
>         readonly attribute float accelerationY; // Acceleration along
>     Y axis in m/s^2
>         readonly attribute float accelerationZ; // Acceleration along
>     Z axis in m/s^2
>     }
>
>     You have to deal with explicit units for a case like this and not
>     clamped/normalized values. What would a normalized offset of 1.0
>     mean? Am I slightly off center? At the other end of the room? It's
>     meaningless without a frame of reference. Same goes
>     for acceleration. You can argue that you can normalize to 1.0 ==
>     9.8 m/s^2 but the accelerometers will happily report values
>     outside that range, and at that point you might as well just
>     report in a standard unit.
>
>     As for things like eye position and such, you'd want to query that
>     separately (no sense in sending it with every device), along with
>     other information about the device capabilities (Screen
>     resolution, FOV, Lens distortion factors, etc, etc.) And you'll
>     want to account for the scenario where there are more than one
>     device connected to the browser.
>
>     Also, if this is going to be a high quality experience you'll want
>     to be able to target rendering to the HMD directly and not rely on
>     OS mirroring to render the image. This is a can of worms in and of
>     itself: How do you reference the display? Can you manipulate a DOM
>     tree on it, or is it limited to WebGL/Canvas2D? If you can render
>     HTML there how do the appropriate distortions get applied, and how
>     do things like depth get communicated? Does this new rendering
>     surface share the same Javascript scope as the page that launched
>     it? If the HMD refreshes at 90hz and your monitor refreshes at
>     60hz, when does requestAnimationFrame fire? These are not simple
>     questions, and need to be considered carefully to make sure that
>     any resulting API is useful.
>
>     Finally, it's worth considering that for a VR experience to be
>     effective it needs to be pretty low latency. Put bluntly: Browser
>     suck at this. Optimizing for scrolling large pages of flat
>     content, text, and images is very different from optimizing for
>     realtime, super low latency I/O. If you were to take an Oculus
>     Rift and plug it into one of the existing browser/Rift demos
>     <https://github.com/Instrument/oculus-bridge> with Chrome, you'll
>     probably find that in the best case the rendering lags behind your
>     head movement by about 4 frames. Even if your code is rendering at
>     a consistent 60hz that means you're seeing ~67ms of lag, which
>     will result in a motion-sickness-inducing "swimming" effect where
>     the world is constantly catching up to your head position. And
>     that's not even taking into account the question of how well
>     Javascript/WebGL can keep up with rendering two high resolution
>     views of a moderately complex scene, something that even modern
>     gaming PCs can struggle with.
>
>     That's an awful lot of work for technology that, right now, does
>     not have a large user base and for which the standards and
>     conventions are still being defined. I think that you'll have a
>     hard time drumming up support for such an API until the technology
>     becomes a little more widespread.
>
>     (Disclaimer: I'm very enthusiastic about current VR research. If I
>     sound negative it's because I'm being practical, not because I
>     don't want to see this happen)
>
>     --Brandon
>
>
>     On Wed, Mar 26, 2014 at 12:34 AM, Brandon Andrews
>     <warcraftthreeft@sbcglobal.net
>     <mailto:warcraftthreeft@sbcglobal.net>> wrote:
>
>         I searched, but I can't find anything relevant in the
>         archives. Since pointer lock is now well supported, I think
>         it's time to begin thinking about virtual reality APIs. Since
>         this is a complex topic I think any spec should start simple.
>         With that I'm proposing we have a discussion on adding a head
>         tracking. This should be very generic with just position and
>         orientation information. So no matter if the data is coming
>         from a webcam, a VR headset, or a pair of glasses with eye
>         tracking in the future the interface would be the same. This
>         event would be similar to mouse move with a high sample rate
>         (which is why in the event the head tracking and eye tracking
>         are in the same event representing a user's total view).
>
>         interface ViewEvent : UIEvent {
>             readonly attribute float roll; // radians, positive is
>         slanting the head to the right
>             readonly attribute float pitch; // radians, positive is
>         looking up
>             readonly attribute float yaw; // radians, positive is
>         looking to the right
>             readonly attribute float offsetX; // offset X from the
>         calibrated center 0 in the range -1 to 1
>             readonly attribute float offsetY; // offset Y from the
>         calibrated center 0 in the range -1 to 1
>             readonly attribute float offsetZ; // offset Z from the
>         calibrated center 0 in the range -1 to 1, and 0 if not supported
>             readonly attribute float leftEyeX; // left eye X position
>         in screen coordinates from -1 to 1 (but not clamped) where 0
>         is the default if not supported
>             readonly attribute float leftEyeY; // left eye Y position
>         in screen coordinates from -1 to 1 (but not clamped) where 0
>         is the default if not supported
>             readonly attribute float rightEyeX; // right eye X
>         position in screen coordinates from -1 to 1 (but not clamped)
>         where 0 is the default if not supported
>             readonly attribute float rightEyeY; // right eye Y
>         position in screen coordinates from -1 to 1 (but not clamped)
>         where 0 is the default if not supported
>         }
>
>         Then like the pointer lock spec the user would be able to
>         request view lock to begin sampling head tracking data from
>         the selected source. There would thus be a view lock change event.
>         (It's not clear how the browser would list which sources to
>         let the user choose from. So if they had a webcam method that
>         the browser offered and an Oculus Rift then both would show
>         and the user would need to choose).
>
>         Now for discussion. Are there any features missing from the
>         proposed head tracking API or features that VR headsets offer
>         that need to be included from the beginning? Also I'm not sure
>         what it should be called. I like "view lock", but it was my
>         first thought so "head tracking" or something else might fit
>         the scope of the problem better.
>
>         Some justifications. The offset and head orientation are self
>         explanatory and calibrated by the device. The eye offsets
>         would be more for a UI that selects or highlights things as
>         the user moves their eyes around. Examples would be a web
>         enabled HUD on VR glasses and a laptop with a precision
>         webcam. The user calibrates with their device software which
>         reports the range (-1, -1) to (1, 1) in screen space. The
>         values are not clamped so the user can look beyond the
>         calibrated ranges. Separate left and right eye values enable
>         precision and versatility since most hardware supporting eye
>         tracking will have raw values for each eye.
>
>
>
>

-- 
Rob

Checkout my new book "Getting started with WebRTC" - it's a 5 star hit
on Amazon http://www.amazon.com/dp/1782166300/?tag=packtpubli-20

CEO & co-founder
http://MOB-labs.com

Chair of the W3C Augmented Web Community Group
http://www.w3.org/community/ar

Invited Expert with the ISO, Khronos Group & W3C
Received on Wednesday, 26 March 2014 19:25:56 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 18:14:22 UTC