Re: Raw Camera Access module - quick update

Hi Piotr,

> On 14 Jul 2022, at 22:32, Piotr Bialecki <bialpio@google.com> wrote:
> 
> > Is your intention to have the current Chrome raw-camera-access implementation available without the feature flag?
> 
> Yes, that's the ultimate goal.
> 
> > I understand the easiest implementation to ship is the one that’s already there, so fully understand if you don’t have the resource or time to commit to any alternatives at the moment, but it would be good to understand the level of interest.
> 
> I think it's unlikely that we'd be able to justify driving the efforts for alternative APIs considering Chrome's current priorities in the AR / XR space. :( My goal with the current proposal is twofold: a) release the API that plenty of people are asking for, and b) don’t close the door for future iterations that would expose this API on devices where the current proposal falls short due to the platform differences.

OK, thanks for the update, makes sense.

> With the above said, I’m personally not opposed to others incubating various approaches, but I’m wondering if a separate session type is necessary here. I think you’ve already been proposing an “inline” session with AR support - would “camera-ar” differ significantly from an inline AR session that has raw camera access enabled on it? What I’m trying to understand here is if this part of the discussion is more about tweaking raw camera access API vs more about either enabling AR for inline sessions or configuring the immersive-ar session to not take over the display on the smartphones. If it’s the latter, then I’d like to still move forward with the current proposal.

My “inline-ar” proposal [ https://github.com/immersive-web/webxr-ar-module/issues/79 <https://github.com/immersive-web/webxr-ar-module/issues/79> ] combined with with raw-camera-access would indeed provide all the needed functionality for our use cases.

When doing computer vision in wasm on the frames (in a worker), we’d need to maintain our own pool of textures and blit the WebXR-provided image there, so we can hold onto it until the vision code is complete before rendering it. We do something similar with gUM video elements already so it’s not the end of the world. The “camera-ar” proposal shifts that texture pool onto the browser side, and exposes the frames to JS as ImageBitmaps. That has some advantages (eg ImageBitmap is transferable, and can be used in multiple contexts) but I could live without it.

When rendering all the pixels, including an older WebXR frame from our texture pool, the implicit compositing of the latest WebXR frame would also feel like overkill. I suppose that could be fixed easily enough by having an API to disable that implicit camera composition in the “inline-ar” spec.

The “inline-ar” approach is the more privacy preserving one (camera implicitly composited, and only available to the page if raw-camera-access is requested) so it does have advantages. It would also support transient input in the same way as the inline session.

The biggest downside is likely implementor interest and effort once again - “inline-ar” doesn’t make any sense for headsets, so Android Chrome seems the most likely candidate for a first implementation (I’d be thrilled to be surprised by Safari here though!). Given what you wrote above about your current priorities, I assume this is also unlikely to be something you’d be able to work on at the moment.

I might attempt to do a prototype of one of these in the Chromium code, and for me the camera-ar one is likely easier as it wouldn’t need to touch the page compositor at all - just getting the ARCore frames in a texture pool and exposing as ImageBitmaps.

So to separate the “implementor interest” a bit - if I was to do some of the implementation work and drive the spec work (for either inline-ar or camera-ar), is there any interest in shipping it?

Thanks!

Simon

> Thanks!
> -Piotr
> 
> 
> 
> On Thu, 14 Jul 2022 at 06:56, Simon Taylor <simon@zappar.com <mailto:simon@zappar.com>> wrote:
> Hi Piotr,
> 
> Thanks for the update, and your efforts to move this one forward!
> 
> Is your intention to have the current Chrome raw-camera-access implementation available without the feature flag?
> 
> I know we had a discussion some months ago at the face-to-face about various ways to expose raw camera access, in brief:
> 
> - Synchronous getter in rAF callback - current raw-camera-access implementation
> - getUserMedia + new constraints - some support in the discussion, but I have some practical concerns
> - A new “camera-ar” session type - would sit in IW’s sphere of influence (unlike gUM) and could share all the pose / anchor / etc functionality
> 
> I discussed the gUM and new session type options in the issue here: https://github.com/immersive-web/webxr-ar-module/issues/78 <https://github.com/immersive-web/webxr-ar-module/issues/78>
> 
> I think someone in the discussion at the f2f (Alex Turner IIRC) suggested writing up some pros/cons for various options on issues. I think I made a decent start on that in that issue, but I’m happy to do more work in that direction if there is value in it and if there is implementor interest.
> 
> I understand the easiest implementation to ship is the one that’s already there, so fully understand if you don’t have the resource or time to commit to any alternatives at the moment, but it would be good to understand the level of interest.
> 
> The camera-ar session feels relatively straightforward an implementation for mobile platforms, would completely address my concerns around giving up layout control to the “immersive” sessions, be easier to polyfill (eg on iOS Safari which lacks full screen atm), would support either “rendering effects” or “computer vision” use cases, and would be implementable on headsets too.
> 
> There’s likely a small performance cost on Android, as you’d need to blit the SurfaceTexture from ARCore to a framebuffer to support the more async access model through an ImageBitmap. I think the performance cost would be worth the benefits, and I believe the ARCore wrappers for various engines do this anyway (ref an old ARCore issue I opened in 2018: https://github.com/google-ar/arcore-android-sdk/issues/668 <https://github.com/google-ar/arcore-android-sdk/issues/668>)
> 
> Simon
> 
>> On 13 Jul 2022, at 18:31, Piotr Bialecki <bialpio@google.com <mailto:bialpio@google.com>> wrote:
>> 
>> Hey Immersive Web!
>> 
>> Just a quick update regarding WebXR's Raw Camera Access module. Latest state is that I've issued a PR <https://github.com/immersive-web/raw-camera-access/pull/12> that should address some concerns from TAG. After that lands, I'm planning on kicking off blink launch process which requires gathering signals from other browser vendors, so don't be surprised if you see some activity there!
>> 
>> Thanks,
>> -Piotr Bialecki
>