Fwd: Intent to Implement: WebXR DOM Overlay for Handheld AR

FYI, here's the intent to implement I had posted this Tuesday. Just to
avoid any misunderstandings, please keep in mind that "intent to implement"
is an early step in the process, see
https://chromium.org/blink/launching-features . Think of it as a
prototyping/design stage, not that a launch is imminent.  Discussions are
still ongoing about using DOM with WebXR.

I'm looking forward to talking about this at TPAC assuming others are
interested also.

---------- Forwarded message ---------
From: Klaus Weidner <klausw@google.com>
Date: Tue, Sep 10, 2019 at 5:05 PM
Subject: Intent to Implement: WebXR DOM Overlay for Handheld AR
To: <blink-dev@chromium.org>
Cc: Mounir Lamouri <mlamouri@google.com>


Contact emails klausw@chromium.org,mlamouri@chromium.org Explainer
https://github.com/immersive-web/dom-overlays Design docs/spec
https://docs.google.com/document/d/e/2PACX-1vRpXB5wX1R1QRzniysT5J1LhLQXAE5OMPX0kQiY-ozv8LsdsP22nf3mDyV6F8G92O_m0qAWMswLqOHT/pub
TAG
review The API surface of this feature is intentionally kept minimal,
essentially it consists of a single WebXR session initialization feature
name that enables use of the existing Fullscreen API while in immersive-ar
mode. Summary This feature allows WebXR applications using immersive-ar on
handheld devices to optionally activate a DOM overlay mode where the 2D
page content is shown as an interactive transparent layer on top of the
application-drawn WebGL content and camera image. It is based on the
existing Fullscreen API which is used to select the visible DOM element.
Motivation Developers want to use DOM to create UI for their XR
experiences. For VR, inline sessions are by definition within the DOM, but
we have deferred this capability for immersive VR sessions. For AR, though,
there is no inline mode, and 2D UI is especially important for popular use
cases. We believe that supporting such UI is part of the minimum viable
product for AR. Risks
Interoperability and Compatibility This is an incubation of a proposed
extension to WebXR within the Immersive Web CG. Support for AR in WebXR is
less mature than VR, so we do not expect any interoperability or
compatibility issues other than with other potential experiments. It's
currently unclear if and how DOM content should be handled for more general
use cases beyond handheld AR, for example standalone AR headsets or desktop
VR. A main motivation for this implementation is to explore the problem
space and inform potential future directions. See for example discussion in
https://github.com/immersive-web/dom-overlays/blob/master/design-choices.md
*Firefox*: No public signals *Edge*: No public signals *Safari*: No public
signals *Web developers*: No signals Ergonomics The feature will be
presented as a part of WebXR Device API. The feature will be most likely
used in tandem with WebGL, and is designed to work on top of the existing
Fullscreen API. Overall the proposed implementation is very similar to the
existing implementation of custom HTML controls for video elements which
uses the same hardware composition mechanisms. Activation As this feature
builds on the WebXR Device API, which is still under development, progress
depends on the continuing maturity and enabling of that API. In addition,
this feature also depends on still to be defined AR capabilities of WebXR.
That said, developers have shown a willingness to experiment with early AR
capabilities. Polyfills would be challenging since that would require
rendering arbitrary DOM content in WebGL. That works for limited subsets,
but is hard to do in general, especially for complex interactions. Security
Similar to Fullscreen API, users may be confused about what is going on.
The feature intentionally follows the example of Fullscreen API. Activation
requires user activation together with a pre-existing consent prompt for AR
sessions, and fully exits fullscreen mode when the immersive AR session
ends.
Debuggability Chrome's Inspector works for the DOM overlay content while in
immersive-ar mode, including screencast, clickable elements, and
interactive style changes. Will this feature be supported on all six Blink
platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
No This feature relies on AR support in the platform. The initial focus is
on Android devices that support ARCore. Is this feature fully tested by
web-platform-tests
<https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md>
? No Tests will be added to WebXR's WPT suite. Link to entry on the Chrome
Platform Status https://www.chromestatus.com/feature/6048666307526656