Re: WebXR Device API call agenda, Apr 2nd 2018

Here are my notes from today's call. As always, all errors are probably
mine and please do reply with corrections.

Trevor: Talked briefly about Firefox Reality PR today:
https://blog.mozilla.org/blog/2018/04/03/mozilla-brings-firefox-augmented-virtual-reality/

Brandon: We have a full agenda today, so let's get started.
Let's start with how we represent rays, which came out of the AR side of
things.
We've been representing them as a full pose, which works out OK for things
like the input model, but it breaks down when we go in the other direction,
putting rays back into the system, constructed by hand. It's easier to give
an origin and direction. Then the only validation is to make sure the
direction is non-zero and normalized.
My feeling is that we should break it into components.

Kip: The fewer members as possible is good, so I think you have the right
direction here.

Brandon: In the absense of other strong opinions, what do we feel is our
preferred representation of a vector? I've been avoid this, but it seems
like we'll need to grapple with it. We could go down the matrix path, with
a three element array. That puts more validation work on us, checking that
there are three elements. We could also break it out into x, y, z in an
interface. Or we could use the DOM Point structure, which is attractive
except that it always felt a little weird, in part because it's a 4D point.
I don't expect that we'll resolve this today, but wanted to get anyone's
temperature.

John P: From a process perspective, should this be in the Issues?

Brandon: I think this is something we can work out in the Issues, perhaps
in the Issue about the rays.

John: Has the cartesion coordinate systems with a unit length of one been
decided on?

Brandon: I think if we went with origin/direction then rather than demand
that it's normalized and throw errors, we could say that we would normalize
it within the browser, which would work unless the direction is 0 length.

Kip: If it's zero length then we could return no hits.

Brandon: I agree. It could be cleaner to return no hit results instead of
throwing. Do we prefer to not break things or to shout that the wrong thing
was passed in? I kind of prefer kind of papering over it.

Kip: I imagine cases where someone has point precision errors, where the
ray is expected to be good.

?: I agree. Normalizing vectors is not always perfect.

Brandon: That's why we should do it in the browser, so that we get the
concept of normalization that we expect.

Kip: A zero length vector will rarely every be actually zero, it will be
some epsilon difference, which we should document.

Brandon: I'll some of these points over into the ray Issue and we can
handle it there.

Kip: As a devil's advocate: Why not quaternions and does the up vector
matter?

Brandon: For one, I tend to like quaternions, but my goodness they are not
an intuitive data structure. The common cases for rays are for doing hit
tests, where the up vector doesn't matter, or I want to shoot a ray down an
axis to intersect with the floor, which will be far easier for devs to
think of a ray than a quaternion.
Do we need an up angle, my gut says no but I'm willing to be wrong. Like
pointing from a controller might need an up vector, but we have the
original pose.

Kip: And we can extend it with an up vector in the future.

?: We will need to specify a left or right handed.

Brandon: We tend to use just the WebGL coordinate system.

?: Let's move this to the Issues just for documentation purposes.

Brandon: Next issue: Should we have controller name strings or not? The
primary concern is when I'm holding a Vive Touch I want to look down and
see a Vive Touch model. The current proposal does not address this. I was
content, but Lewis at MS brought it up as a part of their implementation
for things like see-through AR where you see your hand, you don't want to
render a generic VR model on top of it so we need a way to indicate whether
or not it should be rendered. We could do that with a boolean, but
considering the number of conversations we have around this, it's worth
discussing whether we expose a name for the controller.
There are a number of points on the Issue, including that we should
advertise no named controllers until you are in a session to prevent
fingerprinting. These are better left on the Issue.
I talked with ergonomics people and there's a concern that we'll end up
with people outright rejecting using an app because no recognized
controller is found.
The other issue that might come up, which I'm not terribly concerned about,
where a popular framework has controller models for Touch and Daydream and
nothing else, so browser XYZ advertises all controllers as Touch and
Daydream. We don't want to encourage that behavior.
So, one of the proposals to address that is for us to launch the spec at
the same time as a CDN managed by this group that provides controller
models with names with clear path to contribute new meshes. Then we can
build samples around that and integrated into three and babylon and make it
stupidly easy.

Kip: What can we do to help people working on offline applications? Can we
make it possible to download all of the models?

Brandon: That's an excellent question. Does anyone else have knowledge
about how to handle this?

Kip: Maybe we could offer a reference implementation of a service worker
that serves it up when offline.

Brandon: Some of it will be contextually correct, like a space pistol, but
it seems like the default way would be to download the canonical controller.

Yashar: We could have everyone download every currently available
controller and then use those when offline.

Brandon: We could say "There are the controllers you might use on Windows.
On Android. So controller sets filtered by platform."

Kip: Perhaps a manifest file on the CND that indicates organization.

Brandon: I'm glad that the idea is getting a positive response. This is a
potentially large job. If we're going to go that route, it would be good
for the ecosystem to do.

Ada: If we're going to make the comparison to user agent strings, instead
of giving info about the hardware itself it could give a UUID that matches
exactly to a type of hardware. So, there would be no sub-string matching.

Brandon: I don't necessarily disagree with the premise, but I wonder how it
would be long term. It does make it harder on developers because it's not
human readable. But, if you have people highly motivated to spoof they're
going to look up the string and spoof it. There could be another benefit
that I'm overlooking.

Ada: Could we just provide them with the model itself?

Brandon: We have discussed that in the past. One concern I have is an
entirely selfish one, which is that it's more API surface. Implementation
wise it takes more work for the browser to have the internal cache and
serve it up. Your browser binary just got bigger because you're storing a
cache of controller models. Especially if you could use the lower level
runtime API to get the model.
Oh, and it's unclear how to make it possible to move a sub-section of the
controller.

Ada: What if the browser provides it on a separate layer, on top of the
scene.

Brandon: That is a fair amount of new API. There are technical
complications that I think we've talked about on the past calls. Like, how
does it play nicely with the depth buffer of the scene? Oh, and lighting is
a big issue, too.

Ada: It seems like the kind of thing where a minimal viable product to give
someone a controller until the dev can implement their own.

Kip: I can imagine a category of places where the browser renders
controllers without worrying about depth sorting would work. It raises
questions of whether we'd ever want to change the controller or change the
rendering of the controller that we prompt the user for, like in private
browsing mode?

Brandon: I'm not sure I follow. Maybe in incognito mode maybe we don't
advertise these, maybe there's a default string that indicates that there
is a controller but it's not shared with the page where it might be a
privacy risk. I would not be surprised if there is an Apple WebXR
implementation that only returns an unknown controller string.
Coming from the API surface side, having a string does feel like the
smallest thing we could do to enable that ecosystem to experiment. THere
will surely be a way for third party likes like an a-frame component that
just does the right thing.

Ada: A nice thing about the layers approach is that even if the experience
pauses the user will still have their controllers.

Brandon: For me, it's mostly about being wary of introducing new API.

Kip: I like it quite a bit, but shouldn't be the only way to operate. I'd
like to see more conversation.

Ada: It seems like a fast way to get people started.

Brandon: Ok, let's carry that through to the GitHub Issue. I will
transcribe from the notes. Let's carry on through the topics.
Should anchors are part of the core spec, which was brought up on the AR
call.
So, the concern is that anchors look a lot like our frame of reference
concept and it might make sense to drop FoR and treat them all as anchors
and that's how you deal with placing things in space. Does anyone have
thoughts?

Kip: Any time we can fold two entities into one seems like a good move.

Brandon: I agree. I have heard rumbling from MS in the past that they are
different things, but it seems from recent conversations that we could
represent them as a single structure. My understanding is that we could get
by at the minimum with just calling the FoR an anchor and taking the stage
bounds stuff off of it. We are likely to have multiple APIs that aquire
anchors from different sources. These will all have different methods for
creating an anchor, but then they'll all be treated the same after that.
It's not clear to me how to describe the head model FoR in terms of anchors.

Kip: The idea of moving off stage bounds for other reasons, like there are
variations in how those are expressed so they may need to live in a
different place for other reasons.

Rafael: Anchors are more like points in space, whereas head model are not
something that are "anchored" but can move. I think that's the original
reason why they're considered to be separate.

?: Why can't anchors extend FoRs?

Brandon: Absolute, that was the original intent. The current conversation
is whether there is a reason for them to be separate.

Iker: When you mentioned that anchors are meant to be stationary, but it
depends on the definition we want to give anchors in WebXR. In the
explainer, it says that there could be moving anchors, like when tracking
something that can move like a chair or a marker. So, what is important is
to come up with a specific anchor definition.

Rafael: Ok, but you'll still have situations where you lose tracking and
need to fall back to something like a head model.

Brandon: To me, that feels like getting hung up on verbiage. I think OpenXR
refers to this concept as "spaces". Maybe the concern is that if we call it
an anchor it shouldn't move, then maybe we should call them "anchors" but
there is an existing definition in AR that is useful.
We could have an anchor to the place where my head is when it's held up,
I'm sure that you could define that as an anchor, which feels weird but
mechanically it doesn't feel different than an anchor.

Yashar: Yes, the hangup is whether it is trackable. I think that some
systems actually call it a trackable.

Brandon: Could it be that we want to shuffle the verbiage and have the
concept of a coordinate system or space that is neutral and then from API
like `createAnchorAtPoint` it still returns a coordinate system. So,
everywhere we're using the same base object.

Yashar: Yes, that sounds good.

Brandon: I feel good about that. We'll communicate that back to the AR repo
and see if we can come to a conclusion on that and move forward.

Brandon: There's one thing that people should go look at. The deferred
session item has been outstanding for a long time. We should really tie
that up relatively soon. I want to push on is making sure that we can at
least put in a little framework even if we don't intend to use it right
away. It looks like there is a lot of complexity around navigation, like
how to move from page to page. On the Chrome side and the Firefox side,
this is a concept we want to have, like when I put my phone into my headset
while looking at a headset, we want to have this capability. This is
effectively handled through the user activation concept. We have this in
WebVR and we don't want to regress it, but it's mostly been hung up on
concerns from Nell that pages would be overly reliant on it.
So, I'd like people to review and let's continue the conversation on the
Issue.

Ok, that's it for today. Thanks everyone!

Yashar: It might be useful to record these conversations.

Brandon: We've talked about it in the past *loses mic, mouths "let's
continue this conversation in the next call"*


-- 
Trevor F. Smith
Reality Research Engineer
Emerging Technologies @ Mozilla
trsmith@mozilla.com