W3C home > Mailing lists > Public > public-media-capture@w3.org > July 2014

RE: [depth] depth value encoding proposal

From: Hu, Ningxin <ningxin.hu@intel.com>
Date: Tue, 8 Jul 2014 16:31:36 +0000
To: Benjamin Schwartz <bemasc@google.com>
CC: "public-media-capture@w3.org" <public-media-capture@w3.org>, "Kostiainen, Anssi" <anssi.kostiainen@intel.com>, Rob Manson <roBman@buildar.com>
Message-ID: <1298B14A1D0704468AE73FC92557A1622568D9C1@SHSMSX104.ccr.corp.intel.com>
Hi Benjamin,

Thanks for your comments.

I agree with you. From W3C spec perspective, we should define a natural representation of depth value, such as 16-bits integer. It requires Canvas imageData and WebGL texture to support 16-bits.

This kind of mapping can be a browser implementation detail if possible, for example, used to adapt depth image to existing color-based codec (H.264/VP8) in WebRTC implementation.

What do you think?

Thanks,
-ningxin

From: Benjamin Schwartz [mailto:bemasc@google.com]
Sent: Monday, July 07, 2014 8:01 AM
To: Hu, Ningxin
Cc: public-media-capture@w3.org; Kostiainen, Anssi; Rob Manson
Subject: Re: [depth] depth value encoding proposal

I do not think this kind of arbitrary mapping is appropriate in a W3C standard.  We should arrange for a natural representation instead, with modifications to Canvas and WebGL if necessary.

On Thu, Jul 3, 2014 at 5:16 PM, Hu, Ningxin <ningxin.hu@intel.com<mailto:ningxin.hu@intel.com>> wrote:
Hi,

As working on Media Capture Depth Stream Extension [1], we need to specify the depth value representation of Depth Stream (tracked as issue11 [2]). The precision and range of depth value are important to 3D object scan use case (UC2 in corresponding wiki [3]). To achieve good 3D object scan quality, for example itSeez3D demo [4], it requires at least 16-bits depth representation with 1 millimeter precision.

Native depth camera SDKs usually provides 16bit integer (short) with millimeter unit depth value, such as MS Kinect SDK and Intel Perceptual Computing SDK etc.,
On web platform, Depth Stream should also expose the 16-bits depth value via Canvas. However, the problem is today's either Canvas or WebGL doesn't support 16-bits image format.

One possible solution is to tone-map 16-bits to 8-bits. Chrome's implementation for Tango's depth camera employs this method [5]. it throws lower 4 bits and right shift upper 8 bits. If the raw depth unit is millimeter, it can represent 0-4096mm range and 16mm precision depth value. Tone-mapped 8-bits depth encoding is simple, e.g. the WebGL shader decoder sample is very straightforward [6]. However, our prototype shows it generates 'bricky' point cloud as [7].

Alternatively, another solution is to encode 16-bits into 3x8-bits components (RGB). This paper [8] elaborates this encoding scheme in very details. It can represent 16-bits depth value (0-65535mm range and 1mm precision ) as well as work with existing Canvas/WebGL image format. It also has good quality via video codec which is important for WebRTC usage. Although this encoding is more complex (sample WebGL shader decoder at [9]), we didn't find performance issues when prototyping. 16-bits depth value generates "smooth" point cloud, just as prototype shows [10]

So we propose to use 16-bits to 3x8-bits encoding in Media Capture Depth Stream Extension spec before web platform supports 16-bits image format widely.

Do you have any concerns?

Thanks,
-ningxin

[1] https://github.com/w3c/mediacapture-depth

[2] https://github.com/w3c/mediacapture-depth/issues/11

[3] https://www.w3.org/wiki/Media_Capture_Depth_Stream_Extension

[4] https://sketchfab.com/models/f75614f2297e4ed8bda99e6c24e0d469

[5] https://src.chromium.org/viewvc/chrome?revision=260798&view=revision

[6] https://github.com/huningxin/depthcam_three.js/blob/gh-pages/index.html#L81

[7] https://cloud.githubusercontent.com/assets/1005673/3476322/5eeb18da-02fd-11e4-9a87-16093c76e032.png

[8] http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/depth-streaming.pdf

[9] https://github.com/huningxin/depthcam_three.js/blob/16bits-depth/index.html#L79

[10] https://cloud.githubusercontent.com/assets/1005673/3476336/a288731c-02fd-11e4-847a-f8b95a468eff.png


Received on Tuesday, 8 July 2014 17:06:43 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 16:26:28 UTC