Immersive Web CG meeting minutes 2019-02-12

Minutes are here: https://www.w3.org/2019/02/12-immersive-web-minutes.html and
pasted below.

Immersive Web CG12 Feb 2019Attendees
Presenttrevorfsmith, cwilso, alexis_menard, adarose, alexturn,
Tony_BrainwaiveRegretsChairtrevorfsmithScribecwilso
Contents

   - Topics
   <https://www.w3.org/2019/02/12-immersive-web-minutes.html#agenda>
   - Summary of Action Items
   <https://www.w3.org/2019/02/12-immersive-web-minutes.html#ActionSummary>
   - Summary of Resolutions
   <https://www.w3.org/2019/02/12-immersive-web-minutes.html#ResolutionSummary>

------------------------------

<josh_marinacci> the link in the calendar that you updated on the 8th
doesn't seem to be working

<josh_marinacci> it says the meeting has been canceled or ended

<trevorfsmith> Hmmm. I just used it.

<trevorfsmith> And there are others in it.

<josh_marinacci> /
mit.webex.com/mit/j.php?MTID=mfb8383ef0796cd6999844e1626d7fee6///mit.webex.com/mit/j.php?MTID=mfb8383ef0796cd6999844e1626d7fee6

<josh_marinacci> perhaps my calendar invite hasn't updated. :( could
someone send me the current link?

<trevorfsmith> Please don't paste it into this public log.

<trevorfsmith> Josh, I'll send it hold on.

<adrian> trevorfsmith, I'm also looking for the new link

<blair> me too, trevor. My cal still has the old one

<trevorfsmith> Dang. Ok, everyone who can't get in let me know and I'll DM
you the link

<Nick-8thWall> me

<trevorfsmith> Google calendar is really crap at this.

<adrian> me

scribenick cwilso

<scribe> scribenick: cwilso

trevor: Welcome back!
... we have an agenda:
... subtitles and spatial media (Ada)
... computer vision, issue 4 (Nell)
... 3D rendering and constraints and goals for web extension issue 43
(Trevor)
... lighting estimation repo (Trevor), needs an owner. Any takers?
... If lightning estimation sounds like something you'd like to work on -
write an explainer, etc - please contact Trevor
... Taking up subtitles and spatial media...

Ada: we had a call with the Media and Entertainment Interest Group last
week, they're interested in subtitles in media, and 360 videos are a
popular use of VR.
... one issue they raised is how to do subtitles (where do you place them
relative to the user, who's talking (could be behind the user), how can
they be displayed clearly and crisply...

<Leonard> +q

Ada: in November they opened up a couple of issues in the proposals repo, I
wanted to raise everyone's attention to this issue.

https://github.com/immersive-web/proposals/issues/39

https://github.com/immersive-web/proposals/issues/40

scribe: Please take a look, let's try to get the conversation going.

Leonard: Is this for prerecorded or realtime content?

Ada: They were talking about prerecorded, but live is another great topic.
Could you raise in the issues?

Leonard: would transcription be handled elsewhere?

Ada: Yes, this is just display.

Trevor: Are they doing any thinking beyond spherical video? Subtitles in CG
scenes?

Ada: just video content for the moment. I think CG scenes could also make
use of subtitles, but not sure if it's the same feature. They were focused
on 360 video; I did raise 3D video having other issues (placing subtitles
at infinity, e.g.), but that's not the focus right now.
... this may also tie in with the quad layer/ overlay issue

Trevor: let's take up 3D Rendering in WebExtension XR (aka WEXR): Issue 43

https://github.com/immersive-web/proposals/issues/43

scribe: there seem to be lots of cross-purposes on the rendering side of
this long-lived app system - Blair et al, you'd pointed out that one path
is that XRSession could be the "system" that you hang off of.

<blair> FWIW, for small/simple things, I actually tend to be in the glTF
side, as you describe it.

scribe: but another view is that in order for the browser to exercise a lot
of control and management of 3D items, it might be better to hand off
management of 3D assets to a single system.

<blair> (I can't remember how to queue)

blair: I think both are actually required for different things - for
Holograms in Hololens, and some of the [Landscape apps] in Magic Leap, a
gltf-like solution would make sense.
... gltfs for simple notifications or labels are interesting.
... hologram cube (?) or magic leap prisms - regions of a page - are also
quite interesting.
... I think there's some middle ground here. Being able to render multiple
full-screen apps at the same time is likely not tenable.
... both of those platforms will likely evolve to have much more powerful
top-level UI management.
... this is also quite possibly related to the concept of dioramas.
... https://github.com/immersive-web/proposals/issues/31

cwilso: One more point - we should ensure that whatever model we have for
these "mini-apps", prisms, whatever, they have the ability to interact and
be programmatic, not just declarative, or the model will be very limited.

trevor: yes - may need to do a back-and-forth model where the declarative
system puts in hooks for when the dynamic changes can happen, and call into
dynamic call points.
... would love anyone who's into the graphics side could weigh in, now or
later.

<josh_marinacci> I can't hear kip anymore

<josh_marinacci> too much distortion

kip: I think this is a deep rabbit hole, but has two primary issues: 1) if
you have something running async, you may need to define ways to change
composition. 2) this falls into the same category as defining declarative
things.

<kip> Will summarize here:

<kip> 1: Managing differences in performance between composited contexts

<kip> - What to do when one runs slower and kicks in something like 45hz
space warp

<kip> - What to do when content is slow.. Do overlaid items move
asynchronously or stay frame-locked?

<kip> 2: Rendering intent (eg, realism, non-lit, flat shaded)

<kip> - Lighting models vary greatly depending on if scene is mostly static
or dynamic

Alberto: I wish for the most basic implementation possible, injecting JS
and basic gltf into browser environment. Would be good to kickstart ideas.

<kip> 3: Lighting model (eg, Would PBR + a shared spherical harmonic
encoded Precomputed Radiance Transfer still hold up in 5 years?)

trevor: let's take up computer vision issue

<kip> 4: Alpha transparency. Unable to use depth buffer sorting, limiting
compositing unless moving to declarative content only

blair: I wrote this up a while ago. I'm not sure what Nell wanted to cover
here, but I've talked with folks at Intel and Google about this. The big
question for computer vision is how can we expose camera video inside webXR?
... it's always felt like we should be able to leverage WebRTC, but it has
a few limitations. WebRTC has been talking about adding metadata to
streams, though, like timestamps, maybe even pose etc.
... could work like this: when you open an XR session, camera becomes
available, and RTC streams come out with enough metadata so you could sync.

For smartphone-like devices with one feed, the sync is pretty
straightforward. For other devices like HMDs, you'll need to roughly sync
timestamps, since the framerates will (likely) be different.

scribe: I'm wondering if folks involved in WebRTC would be amenable to
adding this stuff.
... I think we should consider whether this is really where we want to be
adding support - doing CV in JS will likely light CPUs on fire.

(even in WebASM)

scribe: maybe we should look for decimated image pyramids or something like
that, not just frames.

<trevorfsmith> cwilso: I know the WebRTC group here and can help with that
conversation. There are interesting security issues that we'll need to
address.

<trevorfsmith> blair: Like what?

<trevorfsmith> cwilso: Like permissions fatigue.

<trevorfsmith> It would be if users just let anything get access to their
camera feed.

nick: it sounds like people have been worried about complexities. Chris
asked us to think about what would potential use cases. I laid some out in
the issue:https://github.com/immersive-web/proposals/issues/4
... specifically,
https://github.com/immersive-web/proposals/issues/4#issuecomment-461596067

WebXR may want to expose the HMD cameras directly, rather than RTC. You
just need access to the camera feed, interaction to generate hit tests, ...

scribe: for mobile AR, you also need synchronized drawing of CG content
with camera frame.
... I don't know if this last one was part of XRFrame design
... displaying on top of a camera feed from 100ms ago might not be tenable.

<trevorfsmith> cwilso: What we're thinking of as XR cases are really
augmented camera cases.

<trevorfsmith> ... If I'm trying to draw a mustache on people's face it
might be a different category from XR.

blair: I mostly agree with what Nick said - I've gotten a lot of pushback
to adding API surface area to WebXR that somewhat duplicates RTC's API
surface area.
... we didn't do what Nick suggested - syncing camera frames - because of
how we implemented the prototype.
... what Chris said about augmented camera feed - wanting to do something
that works everywhere vs using the platform (e.g. face tracker in
platforms).

Chris: yep, that. There's a current incubation for shape detection to use
the platform for augmented camera feeds -
https://github.com/WICG/shape-detection-api.

Nick: I don't want to preclude the moustache app. But we also want more
powerful use cases.

<trevorfsmith> close the queue

trevor: we're out of time, so let's take it to the issue. I'll ping Nell as
well.
... there was a topic brought up about VR-to-VR navigation; if you get a
chance take a look at that issue.
... thanks all!