Re: [depth] depth value encoding proposal

I don't think getDepthTracks() should return color-video.  If you want to
prototype using depth-to-color mappings, the logical way is to treat the
depth channel as a distinct color video camera, accessible by the usual
device selection mechanisms.


On Tue, Jul 8, 2014 at 9:31 AM, Hu, Ningxin <ningxin.hu@intel.com> wrote:

>  Hi Benjamin,
>
>
>
> Thanks for your comments.
>
>
>
> I agree with you. From W3C spec perspective, we should define a natural
> representation of depth value, such as 16-bits integer. It requires Canvas
> imageData and WebGL texture to support 16-bits.
>
>
>
> This kind of mapping can be a browser implementation detail if possible,
> for example, used to adapt depth image to existing color-based codec
> (H.264/VP8) in WebRTC implementation.
>
>
>
> What do you think?
>
>
>
> Thanks,
>
> -ningxin
>
>
>
> *From:* Benjamin Schwartz [mailto:bemasc@google.com]
> *Sent:* Monday, July 07, 2014 8:01 AM
> *To:* Hu, Ningxin
> *Cc:* public-media-capture@w3.org; Kostiainen, Anssi; Rob Manson
> *Subject:* Re: [depth] depth value encoding proposal
>
>
>
> I do not think this kind of arbitrary mapping is appropriate in a W3C
> standard.  We should arrange for a natural representation instead, with
> modifications to Canvas and WebGL if necessary.
>
>
>
> On Thu, Jul 3, 2014 at 5:16 PM, Hu, Ningxin <ningxin.hu@intel.com> wrote:
>
> Hi,
>
>
>
> As working on Media Capture Depth Stream Extension [1], we need to specify
> the depth value representation of Depth Stream (tracked as issue11 [2]).
> The precision and range of depth value are important to 3D object scan use
> case (UC2 in corresponding wiki [3]). To achieve good 3D object scan
> quality, for example itSeez3D demo [4], it requires at least 16-bits depth
> representation with 1 millimeter precision.
>
>
>
> Native depth camera SDKs usually provides 16bit integer (short) with
> millimeter unit depth value, such as MS Kinect SDK and Intel Perceptual
> Computing SDK etc.,
>
> On web platform, Depth Stream should also expose the 16-bits depth value
> via Canvas. However, the problem is today's either Canvas or WebGL doesn't
> support 16-bits image format.
>
>
>
> One possible solution is to tone-map 16-bits to 8-bits. Chrome's
> implementation for Tango's depth camera employs this method [5]. it throws
> lower 4 bits and right shift upper 8 bits. If the raw depth unit is
> millimeter, it can represent 0-4096mm range and 16mm precision depth value.
> Tone-mapped 8-bits depth encoding is simple, e.g. the WebGL shader decoder
> sample is very straightforward [6]. However, our prototype shows it
> generates 'bricky' point cloud as [7].
>
>
>
> Alternatively, another solution is to encode 16-bits into 3x8-bits
> components (RGB). This paper [8] elaborates this encoding scheme in very
> details. It can represent 16-bits depth value (0-65535mm range and 1mm
> precision ) as well as work with existing Canvas/WebGL image format. It
> also has good quality via video codec which is important for WebRTC usage.
> Although this encoding is more complex (sample WebGL shader decoder at
> [9]), we didn't find performance issues when prototyping. 16-bits depth
> value generates "smooth" point cloud, just as prototype shows [10]
>
>
>
> So we propose to use 16-bits to 3x8-bits encoding in Media Capture Depth
> Stream Extension spec before web platform supports 16-bits image format
> widely.
>
>
>
> Do you have any concerns?
>
>
>
> Thanks,
>
> -ningxin
>
>
>
> [1] https://github.com/w3c/mediacapture-depth
>
> [2] https://github.com/w3c/mediacapture-depth/issues/11
>
> [3] https://www.w3.org/wiki/Media_Capture_Depth_Stream_Extension
>
> [4] https://sketchfab.com/models/f75614f2297e4ed8bda99e6c24e0d469
>
> [5] https://src.chromium.org/viewvc/chrome?revision=260798&view=revision
>
> [6]
> https://github.com/huningxin/depthcam_three.js/blob/gh-pages/index.html#L81
>
> [7]
> https://cloud.githubusercontent.com/assets/1005673/3476322/5eeb18da-02fd-11e4-9a87-16093c76e032.png
>
> [8]
> http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/depth-streaming.pdf
>
> [9]
> https://github.com/huningxin/depthcam_three.js/blob/16bits-depth/index.html#L79
>
> [10]
> https://cloud.githubusercontent.com/assets/1005673/3476336/a288731c-02fd-11e4-847a-f8b95a468eff.png
>
>
>