W3C

Mediastream Image Capture

W3C Editor's Draft 26 April 2013

This version:
http://gmandyam.github.com/image-capture
Latest published version:
http://www.w3.org/TR/image-capture/
Latest editor's draft:
http://gmandyam.github.com/image-capture
Editor:
Giridhar Mandyam, Qualcomm Innovation Center, Inc

Copyright © 2012-2013 W3C® (MIT, ERCIM, Keio, Beihang), All Rights Reserved. W3C liability, trademark and document use rules apply.

Abstract

This document specific the takePhoto() method and corresponding camera settings or use with MediaStreams as defined in Media Capture and Streams [GETUSERMEDIA].

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

Comments on this document are welcomed.

This document was published by the Media Capture Task Force as an Editor's Draft. If you wish to make comments regarding this document, please send them to public-media-capture@w3.org (subscribe, archives). All comments are welcome.

Publication as an Editor's Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

1. Introduction

The API defined in this document taks a valid MediaStream and returns an encoded image in the form of a Blob (as defined in [FILE-API]). The image is provided by the capture device that provides the MediaStream. Moreover, picture-specific settings can be optionally provided as arguments that can be applied to the image being captured.

2. Image Capture API

[Constructor(MediaStreamTrack track)]
interface ImageCapture : EventTarget {
    readonly    attribute PhotoSettingsOptions photoSettingsOptions;
    readonly    attribute MediaStreamTrack     videoStreamTrack;
                attribute EventHandler         onphoto;
                attribute EventHandler         onerror;
                attribute EventHandler         onphotosettingschange;
                attribute EventHandler         onframegrab;
    void setOptions (PhotoSettings? photoSettings);
    void takePhoto ();
    void getFrame ();
};

2.1 Attributes

onerror of type EventHandler,
Register/unregister for Image Capture error events of type ImageCaptureErrorEvent. The handler should expect to get a ImageCaptureError object as its first parameter.
onframegrab of type EventHandler,
Register/unregister for frame capture events of type FrameGrabEvent. The handler should expect to get a FrameGrabEvent object as its first parameter.
onphoto of type EventHandler,
Register/unregister for photo events of type BlobEvent. The handler should expect to get a BlobEvent object as its first parameter.
onphotosettingschange of type EventHandler,
Register/unregister for photo settings change events of type SettingsChangeEvent.
photoSettingsOptions of type PhotoSettingsOptions, readonly
Describes current photo settings
videoStreamTrack of type MediaStreamTrack, readonly
The MediaStream passed into the constructor

2.2 Methods

getFrame
When the getFrame() method of an ImageCapture object is invoked, then if the MediaStreamTrack provided in the contructor does not have the kind attribute set to "video" or if the MediaStreamTrack's readyState is not "live", the UA must fire a ImageCaptureErrorEvent event at the ImageCapture object with a new ImageCaptureError object whose code is set to INVALID_TRACK. If the UA is unable to execute the getFrame() method for any other reason, then the UA must fire a ImageCaptureErrorEvent event at the ImageCapture object with a new ImageCaptureError object whose code is set to FRAME_GRAB_ERROR. Otherwise it must queue a task, using the DOM manipulation task source, that runs the following steps:
  1. Gather data from the MediaStreamTrack into a ImageData object (as defined in [CANVAS-2D]) containing a single still frame in RGBA format. The width and height of the ImageData object are derived from the constraints of the MediaStreamTrack.
  2. Raise a FrameGrabEvent event containing the ImageData to the onframegrab event handler (if specified). {Note: Unlike frameGrabber(), getFrame() returns data only once upon being invoked.}
No parameters.
Return type: void
setOptions
When the setOptions() method of an ImageCapture object is invoked, then then a valid PhotoSettings object must be passed in the method to the ImageCapture object. If the UA can successfully apply the settings, then the UA must fire a SettingsChangeEvent event at the onphotosettingschange event handler (if specified). If the UA cannot successfully apply the settings, then the UA must fire a ImageCaptureErrorEvent at the ImageCapture object whose code is set to SETTINGS_ERROR.
ParameterTypeNullableOptionalDescription
photoSettingsPhotoSettings✔✘
Return type: void
takePhoto
When the takePhoto() method of an ImageCapture object is invoked, then if the MediaStreamTrack provided in the contructor does not have the kind attribute set to "video" or if the MediaStreamTrack's readyState is not "live", the UA must fire a ImageCaptureErrorEvent event at the ImageCapture object with a new ImageCaptureError object whose code is set to INVALID_TRACK. If the UA is unable to execute the takePhoto() method for any other reason (for example, upon invocation of multiple takePhoto() method calls in rapid succession), then the UA must fire a ImageCaptureErrorEvent event at the ImageCapture object with a new ImageCaptureError object whose code is set to PHOTO_ERROR. Otherwise it must queue a task, using the DOM manipulation task source, that runs the following steps:
  1. Gather data from the MediaStreamTrack into a Blob containing a single still image. The method of doing this will depend on the underlying device. Devices may temporarily stop streaming data, reconfigure themselves with the appropriate photo settings, take the photo, and then resume streaming. In this case, the stopping and restarting of streaming should cause mute and unmute events to fire on the Track in question.
  2. Raise a BlobEvent event containing the Blob to the onphoto event handler (if specified).
No parameters.
Return type: void

3. FrameGrabEvent

[Constructor(DOMString type, optional FrameGrabEventInit frameGrabInitDict)]
interface FrameGrabEvent : Event {
    readonly    attribute ImageData imageData;
};

3.1 Attributes

imageData of type ImageData, readonly
Returns an ImageData object whose length and height attributes indicates the dimensions of the captured frame.

3.2 FrameGrabEventInit Dictionary

dictionary FrameGrabEventInit : EventInit {
    ImageData imageData;
};

3.2.1 Dictionary FrameGrabEventInit Members

imageData of type ImageData
An ImageData object containing the data to deliver via this event.

4. ImageCaptureErrorEvent

[Constructor(DOMString type, optional ImageCaptureErrorEventInit imageCaptureErrorInitDict)]
interface ImageCaptureErrorEvent : Event {
    readonly    attribute ImageCaptureError imageCaptureError;
};

4.1 Attributes

imageCaptureError of type ImageCaptureError, readonly
Returns a ImageCaptureError object whose code attribute indicates the type of error occurrence.

4.2 ImageCaptureErrorEventInit Dictionary

dictionary ImageCaptureErrorEventInit : EventInit {
    ImageCaptureError imageCaptureError;
};

4.2.1 Dictionary ImageCaptureErrorEventInit Members

imageCaptureError of type ImageCaptureError
A ImageCaptureError object containing the data to deliver via this event.

5. BlobEvent

[Constructor(DOMString type, optional BlobEventInit blobInitDict)]
interface BlobEvent : Event {
    readonly    attribute Blob data;
};

5.1 Attributes

data of type Blob, readonly
Returns a Blob object whose type attribute indicates the encoding of the blob data. An implementation must return a Blob in a format that is capable of being viewed in an HTML <img> tag.

5.2 BlobEventInit Dictionary

dictionary BlobEventInit : EventInit {
    Blob data;
};

5.2.1 Dictionary BlobEventInit Members

data of type Blob
A Blob object containing the data to deliver via this event.

6. SettingsChangeEvent

[Constructor(DOMString type, optional SettingsChangeEventInit photoSettingsInitDict)]
interface PhotoSettingsEvent : Event {
    readonly    attribute PhotoSettings photoSettings;
};

6.1 Attributes

photoSettings of type PhotoSettings, readonly
Returns a PhotoSettings object whose type attribute indicates the current photo settings.

6.2 SettingsChangeEventInit Dictionary

dictionary SettingsChangeEventInit : EventInit {
    PhotoSettings photoSettings;
};

6.2.1 Dictionary SettingsChangeEventInit Members

photoSettings of type PhotoSettings
A PhotoSettings object containing the data to deliver via this event.

7. ImageCaptureError

The ImageCaptureError object is passed to an onerror event handler of an ImageCapture object if an error occurred when the object was created or any of its methods were invoked.


[NoInterfaceObject]
interface ImageCaptureError {
    const unsigned short FRAME_GRAB_ERROR = 1;
    const unsigned short SETTINGS_ERROR = 2;
    const unsigned short PHOTO_ERROR = 3;
    const unsigned short ERROR_UNKNOWN = 4;
    readonly    attribute unsigned short code;
    readonly    attribute DOMString      message;
};

7.1 Attributes

code of type unsigned short, readonly
The code attribute returns the appropriate code for the error event, derived from the constants defined in the ImageCaptureError interface.
message of type DOMString, readonly
The message attribute must return an error message describing the details of the error encountered.

7.2 Constants

ERROR_UNKNOWN of type unsigned short
An ImageCaptureError object must set its code value to this constant if an error occurred due to indeterminate cause upon invocation of any method of the ImageCapture interface.
FRAME_GRAB_ERROR of type unsigned short
An ImageCaptureError object must set its code value to this constant if an error occurred upon invocation of the getFrame() method of the ImageCapture interface.
PHOTO_ERROR of type unsigned short
An ImageCaptureError object must set its code value to this constant if an error occurred upon invocation of the takePhoto() method of the ImageCapture interface.
SETTINGS_ERROR of type unsigned short
An ImageCaptureError object must set its code value to this constant if an error occurred upon invocation of the setOptions() method of the ImageCapture interface.

8. MediaSettingsRange

interface MediaSettingsRange {
    readonly    attribute unsigned long max;
    readonly    attribute unsigned long min;
    readonly    attribute unsigned long initial;
};

8.1 Attributes

initial of type unsigned long, readonly
The current value of this setting
max of type unsigned long, readonly
The maximum value of this setting
min of type unsigned long, readonly
The minimum value of this setting

9. MediaSettingsItem

The MediaSettingsItem interface is now defined, which allows for a single setting to be managed.

interface MediaSettingsItem {
    readonly    attribute any value;
};

9.1 Attributes

value of type any, readonly
Value of current setting.

10. PhotoSettingsOptions

The PhotoSettingsOptions attribute of the ImageCapture object provides the photo-specific settings options and current settings values. The following definitions are assumed for individual settings and are provided for information purposes:

  1. White balance mode is a setting that cameras use to adjust for different color temperatures. Color temperature is the temperature of background light (measured in Kelvin normally). This setting can also be automatically determined by the implementation. If 'automatic' mode is selected, then the Kelvin setting for White Balance Mode may be overridden. Typical temprature ranges for different modes are provided below:
    Mode Kelvin range
    incandescent 2500-3500
    fluorescent 4000-5000
    warm-fluorescent 5000-5500
    daylight 5500-6500
    cloudy-daylight 6500-8000
    twilight 8000-9000
    shade 9000-10000
  2. Exposure is the amount of light allowed to fall on the photographic medium. Auto-exposure mode is a camera setting where the exposure levels are automatically adjusted by the implementation based on the subject of the photo.
  3. Exposure Compensation is a numeric camera setting that adjusts the exposure level from the current value used by the implementation. This value can be used to bias the exposure level enabled by auto-exposure.
  4. The ISO setting of a camera describes the sensistivity of the camera to light. It is a numeric value, where the lower the value the greater the sensitivity. This setting in most implementations relates to shutter speed, and is sometimes known as the ASA setting.
  5. Red Eye Reduction is a feature in cameras that is designed to limit or prevent the appearance of red pupils ("Red Eye") in photography subjects due prolonged exposure to a camera's flash.
  6. Brightness refers to the numeric camera setting that adjusts the perceived amount of light emitting from the photo object. A higher brightness setting increases the intensity of darker areas in a scene while compressing the intensity of brighter parts of the scene.
  7. Contrast is the numeric camera setting that controls the difference in brightness between light and dark areas in a scene. A higher contrast setting reflects an expansion in the difference in brightness.
  8. Saturation is a numeric camera setting that controls the intensity of color in a scene (i.e. the amount of gray in the scene). Very low saturation levels will result in photo's closer to black-and-white.
  9. Sharpness is a numeric camera setting that controls the intensity of edges in a scene. Higher sharpness settings result in higher edge intensity, while lower settings result in less contrast and blurrier edges (i.e. soft focus).
interface PhotoSettingsOptions {
                attribute MediaSettingsItem  autoWhiteBalanceMode;
                attribute MediaSettingsRange whiteBalanceMode;
                attribute ExposureMode       autoExposureMode;
                attribute MediaSettingsRange exposureCompensation;
                attribute MediaSettingsRange iso;
                attribute MediaSettingsItem  redEyeReduction;
                attribute MediaSettingsRange brightness;
                attribute MediaSettingsRange constrast;
                attribute MediaSettingsRange saturation;
                attribute MediaSettingsRange sharpness;
                attribute MediaSettingsRange imageHeight;
                attribute MediaSettingsRange imageWidth;
};

10.1 Attributes

autoExposureMode of type ExposureMode,
This reflects the current auto exposure mode setting. Values are of type ExposureMode.
autoWhiteBalanceMode of type MediaSettingsItem,
This reflects whether automated White Balance Mode selection is on or off, and is boolean - on is true
brightness of type MediaSettingsRange,
This reflects the current brightness setting of the camera and permitted range. Values are numeric.
constrast of type MediaSettingsRange,
This reflects the current contrast setting of the camera and permitted range. Values are numeric.
exposureCompensation of type MediaSettingsRange,
This reflects the current exposure compensation setting and permitted range. Values are numeric.
imageHeight of type MediaSettingsRange,
This reflects the image height range supported by the UA and the current height setting.
imageWidth of type MediaSettingsRange,
This reflects the image width range supported by the UA and the current width setting.
iso of type MediaSettingsRange,
This reflects the current camera ISO setting and permitted range. Values are numeric.
redEyeReduction of type MediaSettingsItem,
This reflects whether camera red eye reduction is on or off, and is boolean - on is true
saturation of type MediaSettingsRange,
This reflects the current saturation setting of the camera and permitted range. Values are numeric.
sharpness of type MediaSettingsRange,
This reflects the current sharpness setting of the camera and permitted range. Values are numeric.
whiteBalanceMode of type MediaSettingsRange,
This reflects the current white balance mode setting. Values are of type WhiteBalanceModeEnum.

11. ExposureMode

enum ExposureModeEnum {
    "frame-average",
    "center-weighted",
    "spot-metering"
};
Enumeration description
frame-averageAverage of light information from entire scene
center-weightedSensitivity concentrated towards center of viewfinder
spot-meteringSpot-centered weighting

12. PhotoSettings

The PhotoSettings object is optionally passed into the ImageCapture.setOptions() method in order to modify capture device settings specific to still imagery. Each of the attributes in this object are optional.

dictionary PhotoSettings {
    attribute boolean       autoWhiteBalanceMode;
    attribute unsigned long whiteBalanceMode;
    attribute any           autoExposureMode;
    attribute unsigned long exposureCompensation;
    attribute unsigned long iso;
    attribute boolean       redEyeReduction;
    attribute unsigned long brightness;
    attribute unsigned long constrast;
    attribute unsigned long saturation;
    attribute unsigned long sharpness;
    attribute unsigned long imageHeight;
    attribute unsigned long imageWidth;
};

12.1 Dictionary PhotoSettings Members

autoExposureMode of type attribute any
This reflects the desired auto exposure mode setting. Acceptable values are of type ExposureModeEnum.
autoWhiteBalanceMode of type attribute boolean
This reflects whether automatic White Balance Mode selection is desired.
brightness of type attribute unsigned long
This reflects the desired brightness setting of the camera.
constrast of type attribute unsigned long
This reflects the desired contrast setting of the camera.
exposureCompensation of type attribute unsigned long
This reflects the desired exposure compensation setting.
imageHeight of type attribute unsigned long
This reflects the desired image height. The UA must select the closest height value this setting if it supports a discrete set of height options.
imageWidth of type attribute unsigned long
This reflects the desired image width. The UA must select the closest width value this setting if it supports a discrete set of width options.
iso of type attribute unsigned long
This reflects the desired camera ISO setting.
redEyeReduction of type attribute boolean
This reflects whether camera red eye reduction is desired
saturation of type attribute unsigned long
This reflects the desired saturation setting of the camera.
sharpness of type attribute unsigned long
This reflects the desired sharpness setting of the camera.
whiteBalanceMode of type attribute unsigned long
This reflects the desired white balance mode setting.

13. Examples

13.1 Taking a picture if Red Eye Reduction is activated

Example 1
navigator.getUserMedia({video: true}, gotMedia, failedToGetMedia);

function gotMedia(mediastream) {
   //Extract video track.  'kind' attribute not checked because stream was created with video option only.
   var videoDevice = mediastream.getTrackByID()[0];
   // Check if this device supports a picture mode...
   var pictureDevice = new ImageCapture(videoDevice);
   if (pictureDevice) {
         pictureDevice.onphoto = showPicture;
         if (pictureDevice.photoSettingsOptions.redEyeReduction) {
            pictureDevice.setOptions({redEyeReductionSetting:true});
            }
         else
            console.log('No red eye reduction');
         pictureDevice.onphotosettingschange = function(){
            if (pictureDevice.photoSettingsOptions.redEyeReduction.value)
                pictureDevice.takePhoto();
            }
         }
     }

 function showPicture(e) {
    var img = document.querySelector("img");
    img.src = URL.createObjectURL(e.data);
    }

 function failedToGetMedia{
    console.log('Stream failure');
    }

13.2 Grabbing a Frame for Post-Processing

Example 2
navigator.getUserMedia({video: true}, gotMedia, failedToGetMedia);

function gotMedia(mediastream) {
   //Extract video track.  'kind' attribute not checked because stream was created with video option only.
   var videoDevice = mediastream.getTrackByID()[0];
   // Check if this device supports a picture mode...
   var pictureDevice = new ImageCapture(videoDevice);
   if (pictureDevice) {
         pictureDevice.onframegrab = processFrame;
         pictureDevice.getFrame();
         }
     }

 function processFrame(e) {
    imgData = e.imageData;
    width = imgData.width;
    height = imgData.height;
    for (j=3; j < imgData.length; j+=4)
         {
         // Set all alpha values to medium opacity
         imgData.data[j] = 128;
         }
    // Create new ImageObject with the modified pixel values
    var canvas = document.createElement('canvas');
    ctx = canvas.getContext("2d");
    newImg = ctx.createImageData(width,height);
    for (j=0; j < imgData.length; j++)
         {
         newImg.data[j] = imgData.data[j];
         }
    // ... and do something with the modified image ...
    }

 function failedToGetMedia{
    console.log('Stream failure');
    }

A. References

A.1 Normative references

[CANVAS-2D]
Rik Cabanier; Eliot Graff; Jay Munro; Tom Wiltzius; Ian Hickson. HTML Canvas 2D Context. 17 December 2012. W3C Candidate Recommendation. URL: http://www.w3.org/TR/2012/CR-2dcontext-20121217
[FILE-API]
Arun Ranganathan; Jonas Sicking. File API. 25 October 2012. W3C Working Draft. URL: http://www.w3.org/TR/2012/WD-FileAPI-20121025
[GETUSERMEDIA]
D. Burnett; A. Narayanan. Media Capture and Streams. 28 June 2012. W3C Working Draft. URL: http://www.w3.org/TR/2012/WD-mediacapture-streams-20120628/