RE: [depth] depth value encoding proposal

For the spec, I agree with you getDepthTracks() should only return depth stream.

For the prototype, I happened to implement a similar one [1] as you suggest. It appends another color video camera named 'depth' to normal ones and provides tone-mapped 8 bits depth value via Y component of YV12 format. Web apps can use usual device selection mechanism to open it. You can find the demo at [2].

Thanks,
-ningxin

[1] https://codereview.chromium.org/262383003/

[2] https://www.youtube.com/watch?v=7GMkKGPgpx0


From: Benjamin Schwartz [mailto:bemasc@google.com]
Sent: Tuesday, July 08, 2014 9:52 AM
To: Hu, Ningxin
Cc: public-media-capture@w3.org; Kostiainen, Anssi; Rob Manson
Subject: Re: [depth] depth value encoding proposal

I don't think getDepthTracks() should return color-video.  If you want to prototype using depth-to-color mappings, the logical way is to treat the depth channel as a distinct color video camera, accessible by the usual device selection mechanisms.

On Tue, Jul 8, 2014 at 9:31 AM, Hu, Ningxin <ningxin.hu@intel.com<mailto:ningxin.hu@intel.com>> wrote:
Hi Benjamin,

Thanks for your comments.

I agree with you. From W3C spec perspective, we should define a natural representation of depth value, such as 16-bits integer. It requires Canvas imageData and WebGL texture to support 16-bits.

This kind of mapping can be a browser implementation detail if possible, for example, used to adapt depth image to existing color-based codec (H.264/VP8) in WebRTC implementation.

What do you think?

Thanks,
-ningxin

From: Benjamin Schwartz [mailto:bemasc@google.com<mailto:bemasc@google.com>]
Sent: Monday, July 07, 2014 8:01 AM
To: Hu, Ningxin
Cc: public-media-capture@w3.org<mailto:public-media-capture@w3.org>; Kostiainen, Anssi; Rob Manson
Subject: Re: [depth] depth value encoding proposal

I do not think this kind of arbitrary mapping is appropriate in a W3C standard.  We should arrange for a natural representation instead, with modifications to Canvas and WebGL if necessary.

On Thu, Jul 3, 2014 at 5:16 PM, Hu, Ningxin <ningxin.hu@intel.com<mailto:ningxin.hu@intel.com>> wrote:
Hi,

As working on Media Capture Depth Stream Extension [1], we need to specify the depth value representation of Depth Stream (tracked as issue11 [2]). The precision and range of depth value are important to 3D object scan use case (UC2 in corresponding wiki [3]). To achieve good 3D object scan quality, for example itSeez3D demo [4], it requires at least 16-bits depth representation with 1 millimeter precision.

Native depth camera SDKs usually provides 16bit integer (short) with millimeter unit depth value, such as MS Kinect SDK and Intel Perceptual Computing SDK etc.,
On web platform, Depth Stream should also expose the 16-bits depth value via Canvas. However, the problem is today's either Canvas or WebGL doesn't support 16-bits image format.

One possible solution is to tone-map 16-bits to 8-bits. Chrome's implementation for Tango's depth camera employs this method [5]. it throws lower 4 bits and right shift upper 8 bits. If the raw depth unit is millimeter, it can represent 0-4096mm range and 16mm precision depth value. Tone-mapped 8-bits depth encoding is simple, e.g. the WebGL shader decoder sample is very straightforward [6]. However, our prototype shows it generates 'bricky' point cloud as [7].

Alternatively, another solution is to encode 16-bits into 3x8-bits components (RGB). This paper [8] elaborates this encoding scheme in very details. It can represent 16-bits depth value (0-65535mm range and 1mm precision ) as well as work with existing Canvas/WebGL image format. It also has good quality via video codec which is important for WebRTC usage. Although this encoding is more complex (sample WebGL shader decoder at [9]), we didn't find performance issues when prototyping. 16-bits depth value generates "smooth" point cloud, just as prototype shows [10]

So we propose to use 16-bits to 3x8-bits encoding in Media Capture Depth Stream Extension spec before web platform supports 16-bits image format widely.

Do you have any concerns?

Thanks,
-ningxin

[1] https://github.com/w3c/mediacapture-depth

[2] https://github.com/w3c/mediacapture-depth/issues/11

[3] https://www.w3.org/wiki/Media_Capture_Depth_Stream_Extension

[4] https://sketchfab.com/models/f75614f2297e4ed8bda99e6c24e0d469

[5] https://src.chromium.org/viewvc/chrome?revision=260798&view=revision

[6] https://github.com/huningxin/depthcam_three.js/blob/gh-pages/index.html#L81

[7] https://cloud.githubusercontent.com/assets/1005673/3476322/5eeb18da-02fd-11e4-9a87-16093c76e032.png

[8] http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/depth-streaming.pdf

[9] https://github.com/huningxin/depthcam_three.js/blob/16bits-depth/index.html#L79

[10] https://cloud.githubusercontent.com/assets/1005673/3476336/a288731c-02fd-11e4-847a-f8b95a468eff.png