Re: Raw data APIs - 2 - Access to unencoded data

Just want to chime in here;  I’ve been playing with trying to integrate computer vision with the WebXR APIs to support custom sensing in augmented reality contexts, and these proposals here would go a long way toward making that possible:  
- WebXR devices will provide the camera intrinsics, and exposing those is super-important
- raw data access is vital;  all conversions/copies add latency, but also take time/power.  AR tracking/sensing platforms APIs are pretty expensive, so we don’t want to add to it. These APIs are already going to be giving us the video camera image data in a platform specific format (e.g., on iOS it’s a two plane YUV422P format; Android does something different).  Helping the programmer know what the native format it, and letting them chose that or some other format (that they need) seems wise;  the conversions are likely more efficient natively if they are going to do them anyway.  We certainly don’t want to force everything to RGBA32 (like you get out of a canvas)  
- I’m not sure how it fits, but at least on mobile, more and more of the devices have unified memory, so being able to request the data in javascript memory (some raw buffer) and/or in GPU memory (as a WebGL texture) would be very powerful, since this will be very efficient natively

Regarding the timing question:  even if the clocks are fuzzed, knowing (as much as we can) the alignment between the video timestamp and the standard javascript high accuracy clock (such as is passing into the requestAnimationFrame callback, or returned from performance.now()) is what is needed:  sure, it might not be perfect, but knowing how the clocks align as closely as possible is necessary to integrate camera sensing and 3D AR/VR rendering.

I’d think that defining some target for the sensor (e.g., Android’s sensor capture start time) is good, and stating in the standard that its “best effort” (i.e., some platforms may not be able to get it exactly right).



> On May 30, 2018, at 4:01 PM, Stojiljkovic, Aleksandar <aleksandar.stojiljkovic@intel.com> wrote:
> 
> Fair enough, both are valid points. I thought it was good to reopen timestamp approach.
> 
> For the reference - clock resolution security issues [1].
> 
> Android camera defines the timestamp as sensor capture start time [2]. Note that SENSOR_INFO_TIMESTAMP_SOURCE could have value UNKNOWN and we would get monotonic time, that cannot be compared to timestamps from other subsystems, e.g. sensors.
> 
> [1]
> https://www.w3.org/TR/hr-time/#clock-resolution <https://www.w3.org/TR/hr-time/#clock-resolution.>
> 
> [2]
> https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics#SENSOR_INFO_TIMESTAMP_SOURCE <https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics#SENSOR_INFO_TIMESTAMP_SOURCE>
> 
> Kind Regards,
> Aleksandar
> 
> From: Randell Jesup [randell-ietf@jesup.org]
> Sent: Wednesday, May 30, 2018 7:53 PM
> To: public-webrtc@w3.org
> Subject: Re: Raw data APIs - 2 - Access to unencoded data
> 
> On 5/30/2018 8:10 AM, Harald Alvestrand wrote:
>> Den 30. mai 2018 13:42, skrev Stojiljkovic, Aleksandar:
>>>> Interface Buffer ... Long long timestamp; // for start of buffer. Definition TBD.
>>> Maybe DOMHighResTimeStamp
>>> <https://www.w3.org/TR/hr-time/#sec-domhighrestimestamp> <https://www.w3.org/TR/hr-time/#sec-domhighrestimestamp>, to enable sync
>>> with other events, e.g. Sensor interface
>>> <https://www.w3.org/TR/generic-sensor/#the-sensor-interface> <https://www.w3.org/TR/generic-sensor/#the-sensor-interface>.
>> DOMHighResTimeStamp is convenient for absolute times, when it's clear
>> what it's related to (time the light hit the camera sensor?). For
>> playback of stored media, we might want to use a clock relative to the
>> start of the media; I know there are various well known apps for doing
>> things like time-stretching of videos - I'm not sure how that would be
>> representable in an API.
> 
> I'll note that any use of high-resolution timers by apps may be constrained by the wonders of Spectre mitigation for some time.   It would be nice to un-fuzz the timers, but even once one  has implemented fission/site-isolation/whatever-edge-will-call-it, lots of high-res clock sources make it super-easy to do spectre attacks. :-(
> 
> 
> -- 
> Randell Jesup -- rjesup a t mozilla d o t com
> Please please please don't email randell-ietf@jesup.org <mailto:randell-ietf@jesup.org>!  Way too much spam
> ---------------------------------------------------------------------
> Intel Finland Oy
> Registered Address: PL 281, 00181 Helsinki 
> Business Identity Code: 0357606 - 4 
> Domiciled in Helsinki
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.