[mediacapture-depth] Editor's Draft updated, call for review

Hi All,

The mediacapture-depth spec editors received feedback from implementers that the way how the latest published WD spec extends the list of image sources for 2D rendering contexts would cause intrusive changes to the implementation. This indirect path did work for the experimental implementation, but was perhaps an unintuitive one: the depth stream was proxied via a video element to a canvas element using .drawImage(), and the depth data was read back from canvas via its data attribute.

To address this issue, we've updated the Editor's Draft spec [1] and introduced new purpose-built interfaces for repeated grabbing of frames that give direct access to ImageData and newly minted DepthMap. These additions will deprecate the indirect path.

All - please let us know if you have any concerns with this direction.

For a quick overview of what these changes mean from the web developer's point of view, please compare the examples that are practically functionally equivalent:

* example using the old API [2] (indirect path using video and canvas elements)
* example using the new API [3] (new FrameGrabber, FrameData, DepthMap)

If we hear no concerns we'll continue iterate on the new API, and fill in the blanks in the ED spec.

Here's the summary of changes:

* Removed the video and canvas element indirection: a depth video element may no longer be used as a CanvasImageSource i.e. passed as an argument to drawImage().

* Removed depth specific extensions from the ImageData interface.

* Added a DepthMap interface that has a shape similar to that of ImageData but represents a depth map instead. The depth pixels (aka dexels) in the array have a unit that defaults to millimeters, represent the range [0, 65535] in millimeters in millimeter granularity (i.e. the theoretical range is up to 65.5 metres).

[Should we allow the developer to configure the unit? Based on the public data collected Uint16Array with millimeter units maps pretty nicely to the current 3D cameras (Kinect, Tango, RealSense). Optionally, using meter unit with Float32Array as the data type would provide future extensibility. We have some extension points baked in the spec for this already.]

* Added a FrameData interface that encapsulates ImageData and DepthData, provides timestamps for both, as well as the current time, to enable correlating the frames.

* Added a FrameGrabber interface that takes a MediaStream, hands out FrameData objects via a callback that is invoked periodically. The developer may configure the interval.

[We initially looked at reusing mediacapture-image for this, but felt that would have required us to do drastic changes to that API. Also, we felt the mediacapture-image spec was designed to address a different set of requirements. In particular, reliance on the rAF or setInterval for repeated grabbing of frames was seen problematic from the mediacapture-depth point of view.]

* Extended MediaTrackConstraints with unit (we spec'd this as a partial dictionary since that seemed the most obvious way to do so).

* Extended MediaTrackSettings with unit, near, and far values that provide the web developer information on the minimum and maximum sensing distance of the depth camera.

It should be noted that the new additions have not been spec'd in their entirety to allow us to iterate on the spec quickly based on feedback from the group. In particular, the algorithms and prose around some of the newly minted interfaces is missing -- the WebIDL and examples [3] should give you an idea of the API shape though.

Thanks,

-Anssi (on behalf of the editors)

[1] http://w3c.github.io/mediacapture-depth/
[2] http://www.w3.org/TR/2015/WD-mediacapture-depth-20150129/#examples
[3] http://w3c.github.io/mediacapture-depth/#examples

Received on Wednesday, 13 May 2015 11:06:00 UTC