WebXR - AR Focus Meeting Notes 2018/05/22

*Here are my meeting notes. I wasn't able to capture all of the
conversation that I was engaged in as it was hard to type and talk. I
linked some new issues in the hit-test repo at the bottom. Thanks for the
great discussion!--------Trevor: Asks about existing proposalsMax: Hit-test
has been going smoothly - no major comments, replace README.md with
explainer? Maybe a meta-issue for all proposals? Our implementation has
been going smoothly.Iker: Some discussions several weeks ago but no
specific proposals. Some more forward looking discussions about trackables
but no specific IDL generated yet.Trevor: any further discussion about
coordinate systems, naming, or anchors?Brandon: it was not discussed at
last week's meeting.Trevor: We can discuss whether we have consensus about
this with wider group or if we are ok with leaving it alone as we go
forward with standardization. Brandon, any thoughts?Brandon: The solution
is that this seemed to be an issue of documentation more than anything
else. The primary concern is we have the XR Coordinate system which was
intended to be an anchor-like system. We didn't want people to get into a
mode where people designed their applications in a way that an anchor
concept would be difficult to retro-fit. After trying to squash down frame
of reference in a way that it could act like an anchor, we (specifically
microsoft) thought frame-of-reference was fundamentally different
assumptions - head-space, lost tracking and still want to present something
to the user head-locked, there isn't a clean way to communicate that
through anchors and still retain the meaning of what an anchor is. What
that came down to was we can keep the frame-of-reference concept distinct,
but we need to ensure that when we document and put together samples, there
is a clear expectation that coordinate systems are not a static
never-changing thing, they are something you have to query/monitor and they
could shift in the world. As long as we can do expectation-setting that
frame-of-references won't be the only kind of coordinate system out there.
We don't want to give the impression that you can assume a global
coordinate system for everything. As long as we can skip that assumption,
we don't need to collapse everything down to a single coordinate
type.Trevor: turn it over to Blair so he can discuss some issues about
anchors / hit-testBlair: I have been doing a bunch of little experiments
with APIs and how they work. I think the hit-testing and anchors and think
they are proceeding reasonably - one thing I would bring up is we need to
be careful to e. Admittedly this is in my iOS app that is similar to the
Google WebARonAR app which has an extreme asynchronous nature that the
native implementations won't have. I know in the hit-testing there was a
discussion of guarantee of when it is guarantee that it will resolve in
next RAF or is it a best-effort thing? This has gone back and forth, the
discussions seemed to imply people were migrating towards trying to
guarantee it will be resolved in the next frame. That will prevent things
that do polyfills and such from being able to conform to the spec in
non-native ways because of long-communication times. The reason this is
important is when you get a result back there is a race condition - we
don't have a global coordinate system and the underlying anchor api could
adjust the coordinate systems on any frame and the return values could be
wrong if they are one-frame off. They could end up being from the previous
frame for before the world shifted.Max: describes how ARCore/Chrome
worksBlair: that sounds good, seems like that approach will work on systems
where there is a long round-tripIker: we would be interested in hearing
from others if those constraints workYashar: On MSFT on the hololens side,
we thought with native APIs you load up the mesh and do your own raycasts.
You could imagine the engine doing the same thing, our fault position is
the opposite - we are trying to prove to ourselves that it can't be
synchronous. For example, we noticed that in ARCore ARFrame's hit-test
method is synchronous - it would return a hit-result immediately. It would
be the responsibility of the runtime to be efficient there.Blair: today, I
converted the samples I have to work with WebXR for the asynchronous thing
- it is a bit of a pain-in-the-ass in terms of how you implemented it. We
used to have a synchronous hit-test that uses the in-memory copies of the
planes with ARKit. That is cleaner with what you would expect, on the other
hand - I wanted to do a reticle, do the hit-test, resolves a promise, as
part of that hit-test promise, do another hit-test - it's almost like
another RAF. I agree it's not what someone coming from native - ARKit -
would think to do immediately.Kearwood: question for hardware vendors /
native API definers - would there be any discrepancy if any meshes in the
world or positions in of hits changed in between when the query was formed
and when it was evaluated - 3 events - request, when tested, and when
resolved. Is the information reprojected or invalid if you wait too
long?Iker: In the case of ARCore, the whole spec, if you are waiting more
than one frame there could be a change in the world. It is not common, but
it will happen. What we are trying to assume, even though I agree it's not
the way native developers are used to doing this, what we have to ensure is
that the data you are getting is accurate. The RAF you are getting - the
information of the pose, state of the world, and hit-test promise
resolution should all be accurate. Because it is asynchronous, changes
between when you make the request could happen.Kearwood: let's describe the
variability that people should be accounting for when using the spec -
describe this in the spec text. Would we need to go further? in the result
information provide details of what frame it is valid for or any rear
projection.Blair: if I am understanding what you are saying, this is
already handled in the text - the text basically says you do the hit test,
the results in the promise are valid for the rest of the data you get in
the next RAF. If in the next raf you are asking for the positions of the
anchors in world coordinates, the results of that hit-test are in that same
coordinate system. It would be as if you already had an anchor and said
"where is that"? Max: Yes, exactly. The expected idiom is you do a hit-test
and then place a new object in the then block of the result promise. That
object is rendered for the first time in the subsequent RAF callback and
will be at a perfectly accurate location at that point since the hit-test
result was valid for the world state as of that frame. If you also
establish an anchor in that same then block, it would be updated before the
RAF after the next one.Blair: How do anchors work? are they async?Iker: We
haven't really figured this out. Whether anchors are sync or async is
Blair: based on what you just said, as long as we are explicit in the
anchor text, it could have the same sequence as hit-test. Any operations on
anchors outside of the RAF need to correspond to the coordinate system of
the next raf. If people keep data around from a RAF and try to use it, it
might be invalid on the next RAF. The sequence you described is fine, but
if the USEr had some async task running doing computations in world
coordinates using data from the previous raf, they can't do operations on
anchors without potentially being off in some way.Iker: it could be nice to
have timestamps, lots of AR systems have timestampsJohn: question for Iker
and Max: is there a situation where the hit-test could be starved?Max: yes,
could be starved if not rate-limitedBlair: rate-limiting is fine. My
suggestion is any given platform is free to limit. We should put something
in the spec that we just start rejecting promises when the platform decides
you have made too many. In general, we should make the spec so we don't
allow you to fall behind if you make too many requests. No point in even
supporting that.Max/John: 2d hit test?Blair: the 2d thing is weird, ARKit
is 2d right now, I guess I am not so concerned with the kind of example you
listed - I have ben doing the async 2d hit-test in my test - if you can
move the camera that fast relative to the frame-rate, you probably have
other problems with user-perception, nausea, death? It feels like something
that could happen but so what? The browser will do its best effort. In my
30-60hz tests, you can't even notice the latency.Yahsar: for placement/room
analysis, not a huge issue. If you are placing a reticle and getitng
jitter, the latency is going to be very noticeable. The native API Iker:
you propose sync?Yashar: or async but it would come back in the same frame
if you ask it - you would have to go out of your way to not get it within
the same frame. Walk into a wall?Blair: We need to separate out hit-testing
and world-querying that is different from the world to one-frame
hit-testing that would be used to place objects / reticle. There are
anchors/hit-test that are low-world knowledge thing that people can run on
the web without feeling like they are giving up too much information. There
is going to be a jump where the web-page can have world-knowledge - planes,
meshes, etc. Then there is full camera access where the app can analyze the
environment completely. I think if you were going to write a hololens game
- you ask for world knowledge and the user says "no", you don't get this
information. The app has to adjust and say Ok, I will adjust or I wont run.
That pathing behavior that requires the mesh probably won't work in the
case where the world understanding is denied. We need to separate these
types of experiences.Yashar: yeah to the previous point, the intraframe
hit-test RAF, will it makes sense that platforms can do a sync hit-test
within the frame without slowing things down. If we can make a sensible
pattern that doesn't disturb apps in that "first wave" that you were
describing, then I would be comfortable with that.Blair: my perspective is
being a fly on the wall and talking with people dealing with the actual
implementations. My sense is the overhead of going all the way through
these levels from javascript all the way down to native and back, adding in
a bunch of sync calls that slow things down is potentially a bad
idea.Kearwood: I would want to quantify that. A promise queue is very
implementation specific. What might be worth discussing is a notion of
batching - can we make a single request? It wouldn't just be reducing the
overhead for internal wiring of the browser, but could also be some benefit
in it guaranteeing a batch set of requests that are resolved in
coordination. You are only interested in them being resolved in
coordination - based on the same world state? Can we explore the notion on
batched requests? Would reduce the per-call overhead.Iker: I would suggest
making a proposal - seems like an idea that has merit.John: I appreciate
the delineation of use-cases - reticle vs. scene understanding - both could
be done with hit-test but also both could be done with batching.Max: I will
add these issues to the hit-test repo.Trevor: opening up the floor. Any
other things?Iker: I think it will be hard to do hit-test synchronously in
the web.John: I want to share that we did demonstrate our implementation of
the hit-test API at I/O a couple weeks ago as well as show developers some
sample code. The goal was to get developers engaged and get some feedback.
We had some interesting conversations with people and they had some new
ideas about how to use this stuff. Blair: When we initially implemented the
polyfill last fall, we did the async model, but as soon as people tried to
integrate with a-frame / three, we did some of that using the in-memory
stuff but it wasn't necessary. They understood it better and it It fit
better with the current code they had, those are not bad things, but given
that we are evolving three.js and other platforms to go with the evolving
shape of WebXR, much of which is async, we can evolve the set of utilities
and patterns to match that.Max: to summarize, the suggested options for
hit-test API are - promise that resolves prior to RAF for which it is valid
(current spec)- synchronous call (most likely batched)- 2D hit-test (for
pixel-perfect reticles) TODO: open issue about pixel-perfect reticlesBlair:
we could always expose the world understanding to JS, like we did with
WebARonARKit/Core, and do the raycast in JS or we could do it at a layer
just beneath the surface by piping all the world understanding data up from
the AR system.Max: problems with that include privacy/security of sending
the data up there, code liability of having to reimplement hit-test
algorithms - mismatch of results as bugs are fixed/etc and not necessarily
having all the world elements included.Yashar: Part of having the guarantee
of letting apps do the request at the beginning of the frame and get
results synchronously is that apps can have simple code that is more
aligned with its native counterpart. if that is too heavy of a lift to
bring all the world understanding close enough to the surface of the
browser to have it quickly accessible, which I understand could depend
heavily on the specifics of the browser's code, let's just bucket it by
use-case and enable folks to do what they want.Blair: the longer term
stuff, I would like to start writing it down and discuss it - the
geospatial stuff and the computer vision stuff - been playing with them. I
have a firmer image in my mind of what those things mean - unless there are
objections, i would like to create repos for them and start putting
commentary there so there is a place to discuss them. I think the
geospatial stuff in pretty quick but the computer vision stuff is much more
heavy weight and media capture and web RTC folks need to chime in. I think
it would be worth putting that stuff in there so other people can see it an
contribute - if nobody has any objections, I would like to create repos for
these two issues. At a high-level, I think we can support geospatial AR on
platforms that have compasses by simply knowing whether you can align the
orientation of the coordinate system with geospatial stuff. Knowing you
can't align them means you can't do it, that is also useful knowledge. New
hit-test Issues: - batching:
https://github.com/immersive-web/hit-test/issues/19
<https://github.com/immersive-web/hit-test/issues/19>- sync call support:
https://github.com/immersive-web/hit-test/issues/20
<https://github.com/immersive-web/hit-test/issues/20>- example of best
practices: https://github.com/immersive-web/hit-test/issues/21
<https://github.com/immersive-web/hit-test/issues/21>- reticle support
(existing): https://github.com/immersive-web/hit-test/issues/4
<https://github.com/immersive-web/hit-test/issues/4> - 2d hit-test
(existing): https://github.com/immersive-web/hit-test/issues/7
<https://github.com/immersive-web/hit-test/issues/7> *