Constraints and capabilities: revised proposal

Group,

Here is a revised proposal for constraints and capabilities.  Roughly speaking, it includes the following changes:

- completed IDL for capabilities (after correcting JS mistakes in the structure)
- constraint satisfaction algorithm now defines how browsers must treat unsupported constraints
- changed constraint name syntax from hyphens to camelCase to work better with JS
- added more examples
- added explanations of constraint examples
- fleshed out Capabilities section a bit more
- added proposal for what getCapabilities() should return in the untrusted scenario
- added section defining which constraints are mandatory to support
- added constraint registration section defining the two constraints that seemed to be of most interest
- added examples of some other constraints we could consider defining.  I'd love to hear which ones are of interest to people in this group.

-- dan


=========
= WebIDL =
=========

interface NavigatorUserMedia {
 void getUserMedia (MediaStreamConstraints? constraints, NavigatorUserMediaSuccessCallback? successCallback, NavigatorUserMediaErrorCallback? errorCallback);
 MediaStreamDeviceCapabilitiesArray getCapabilities();
};

dictionary MediaStreamConstraints {
 sequence<MediaStreamConstraint>? mandatory = null;
 sequence<MediaStreamConstraint>? optional = null;
};

interface MediaStreamDeviceCapabilities {
  attribute DOMString id;
  attribute MediaStreamConstraintSet? constraints=null;
}

A MediaStreamConstraint is a dictionary containing exactly one key-value pair, where the key MUST be a valid registered constraint name in the IANA-hosted RTCWeb Media Constraints registry [1] and the value SHOULD be as defined in the associated reference[s] given in the registry.
A MediaStreamConstraintSet is a dictionary containing one or more key-value pairs, where each key MUST be a valid registered constraint name in the IANA-hosted RTCWeb Media Constraints registry [1] and its value SHOULD be as defined in the associated reference[s] given in the registry.

============
= Constraints =
============

When given constraints in the getUserMedia() call, the browser MUST conform to the semantics of the following algorithm when selecting the media stream(s):

1. Define the following:
 a. the set of media types (at the time of writing, only "audio" and "video" are available) addressed by the constraints to be the RequestedMediaTypes.
 b. the FinalSet to be (initially) empty.
 c. the FirstPassSet to be all possible streams the browser can return.
2. For each constraint key-value pair in the mandatory sequence, in order,
 a. If the constraint is not supported by the browser, call the errorCallback with an error indicating failure to satisfy the current mandatory constraint and naming the constraint, then exit.
 b. Remove from the FirstPassSet any streams of the constraint's media type that cannot satisfy the value given for the constraint.
 c. If the FirstPassSet no longer contains at least one stream for every RequestedMediaType, call the errorCallback with an error indicating failure to satisfy the current mandatory constraint and naming the constraint, then exit.  Otherwise, continue with the next mandatory constraint.
3. For each media type in the RequestedMediaTypes,
 a. Define the CandidateSet to be the subset of the FirstPassSet that match the current media type.
 b. Define the SecondPassSet to be the current contents of the CandidateSet.
 c. For each constraint key-value pair in the optional sequence that are for the current media type, in order,
   1. If the constraint is not supported by the browser, skip it and continue with the next constraint.
   2. Remove from the SecondPassSet any streams that cannot satisfy the value given for the constraint.
   3. If the SecondPassSet is now empty, define the SecondPassSet to be the current contents of the Candidate Set.  Otherwise, define the CandidateSet to be the current contents of the SecondPassSet.
 d. Select one stream from the CandidateSet to add to the FinalSet. 
4. Call the successCallback with the list of streams in the FinalSet.

In step 3d the decision of which stream to choose from the CandidateSet is completely up to the browser.  Unless and until a new set of constraints is provided, the browser MAY change its choice of stream at any point, provided that it notifies the application code by again calling the successCallback with the resulting new FinalSet.  It may wish to do this, for example, if the user interface or network congestion changes.

Example:

navigator.getUserMedia(
 {mandatory:[{videoMinHeight:600}, {videoMaxBandwidth:500}],
   optional:[
     {videoMaxAspectratio:1.333333333333},
     {videoMinTimebetweenrefframes:20},
     {videoMinFramerate:30},
     {videoEnumAutowhitebalance:on}]},
 gotStream, didntGetStream);

This example requests a video stream with the properties shown.  Note that if the browser can satisfy either a minimum time between reference frames of 20 or a minimum framerate of 30 but not both, then it will satisfy the former.  Also, if the browser cannot, at the least, provide a video stream with a minimum height of 600 and a maximum bandwidth of 500, then the call to getUserMedia will fail.


Example:

navigator.getUserMedia(
 {mandatory:[{videoEnumProvide:true}],
  optional:[{audioEnumProvide:true}]},
 gotStream, didntGetStream);

This example requests both an audio and a video stream.  If the browser cannot, at the least, provide a video stream, then the errorCallback will be called with an error.


Example:

navigator.getUserMedia(
  {mandatory:[{videoEnumProvide:true}, {audioEnumProvide:true}]},
 gotStream, didntGetStream);

This example requires that the browser either return a stream with both audio and video or call the errorCallback with an error.


============
= Capabilities =
============

The return value from getCapabilities() is a JavaScript Array containing, for each device/channel, an id and a dictionary consisting of the MediaStreamConstraint keys and values that can be satisfied by the device/channel.

The id MUST be unique relative to all other ids in the same MediaStreamDeviceCapabilities array and MUST be composed of the high-level media type returnable by the device (at the time of writing, only "video" and "audio" are valid types), followed by an opaque alphanumeric string.  The purpose of the id is to distinguish among devices of the same media type without providing privacy-revealing information.

A constraint is "supported" if its meaning and behavior are supported as described in the RTCWeb Media Constraints registry [1].
A constraint is "satisfiable" if there is at least one stream that can be produced which satisfies the constraint as described in the RTCWeb Media Constraints registry [1].

=== Trusted scenario ===

In the trusted scenario, in the MediaStreamConstraintSet provided by the browser for each device/channel, the browser MUST return all supported satisfiable constraints of type "min" or "max" as follows:
- if the constraint is a "min" constraint, it must represent the smallest value of that constraint across all supported streams of the given stream type.  For example, if the constraint is "video-min-width:500", that would mean that there is no video stream with a width smaller than 500 pixels.
- if the constraint is a "max" constraint, it must represent the largest value of that constraint across all supported streams of the given stream type.  For example, if the constraint is "video-max-width:1000", that would mean that there is no video stream with a width greater than 1000 pixels.

Example: 

[{id:camera001,
  capabilities:{
 videoEnumProvide:  true,
 videoMinWidth:  800,
 videoMaxWidth:  1024,
 videoMinHeight:  600,
 videoMaxHeight:  768,
 videoMinAspectratio:  1.333333333333,
 videoMaxAspectratio:  1.333333333333,
 videoMinFramerate:  24,
 videoMaxFramerate:  60,
 videoMinPixelrate:  15,
 videoMaxPixelrate:  47,
 videoMinTimebetweenkeyframes;  20,
 videoMaxTimebetweenkeyframes:  40,
 videoMinBandwidth:  1.5,
 videoMaxBandwidth:  3.5}},
 {id:camera002,
  capabilities:{
 videoEnumProvide:  true,
 videoMinWidth:  1600,
 videoMaxWidth:  1920,
 videoMinHeight:  1080,
 videoMaxHeight:  1200,
 videoMinAspectratio:  1.33333333333,
 videoMaxAspectratio:  1.77777777777,
 videoMinFramerate:  24,
 videoMaxFramerate:  120,
 videoMinPixelrate:  57.6,
 videoMaxPixelrate:  248,
 videoMinTimebetweenkeyframes;  20,
 videoMaxTimebetweenkeyframes:  40,
 videoMinBandwidth:  8,
 videoMaxBandwidth:  29.4}},
 {id:audio001,
  capabilities:{
 audioEnumProvide:  true,
 audioMinBandwidth:  1.4,
 audioMaxBandwidth:  128,
 audioMinMos:  2,
 audioMaxMos:  5,
 audioMinCodinglatency:  10,
 audioMaxCodinglatency:  50,
 audioMinSamplingrate:  8000,
 audioMaxSamplingrate:  48000}}]


=== Untrusted scenario ===

In the untrusted scenario, every MediaStreamConstraintSet provided by the browser for each device/channel must be null.

Example:

[{id:camera001},
 {id:camera002},
 {id:audio001}]

The specification of how to determine whether the current application is "trusted" or "untrusted" is TBD.


==================
= Constraint Support =
==================

Support for the following constraints is REQUIRED:  videoEnumProvide, audioEnumProvide.


=====================
= Constraint Registration =
=====================

This specification requests that IANA register the following constraints in the RTCWeb Media Constraints registry [1]:

1. videoEnumProvide
- The constraint name to be registered:  "videoEnumProvide"
- Name and email address of a contact person for this registration:  Daniel Burnett <dburnett@voxeo.com>
- Organization or individuals having the change control:  W3C WebRTC Working Group
- Reference(s) to the specification(s) defining the constraint:  <<<< This document >>>>

Definition of this constraint:

Allowed values for this constraint are:  "true".
When returned as a capability:  the browser MUST return the value "true" if and only if there is at least one device for which a video stream can be returned.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream.


2. audioEnumProvide
- The constraint name to be registered:  "audioEnumProvide"
- Name and email address of a contact person for this registration:  Daniel Burnett <dburnett@voxeo.com>
- Organization or individuals having the change control:  W3C WebRTC Working Group
- Reference(s) to the specification(s) defining the constraint:  <<<< This document >>>>

Definition of this constraint:

Allowed values for this constraint are:  "true".
When returned as a capability:  the browser MUST return the value "true" if and only if there is at least one device for which an audio stream can be returned.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any audio stream.




[1] http://tools.ietf.org/html/draft-burnett-rtcweb-constraints-registry-01


*********************************************************************************************
*** Here are some examples of other constraints we could define.  They are 
*** ONLY EXAMPLES.
*********************************************************************************************
3.  videoMinWidth
Definition of this constraint:

Units for this constraint are:  pixels.
When returned as a capability:  the browser MUST return the minimum width, in pixels, of any resolution the video device is capable of providing.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream whose width is greater than or equal to the value provided.


4.  videoMaxWidth
Definition of this constraint:

Units for this constraint are:  pixels.
When returned as a capability:  the browser MUST return the maximum width, in pixels, of any resolution the video device is capable of providing.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream whose width is less than or equal to the value provided.


5.  videoMinHeight
Definition of this constraint:

Units for this constraint are:  pixels.
When returned as a capability:  the browser MUST return the minimum height, in pixels, of any resolution the video device is capable of providing.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream whose height is greater than or equal to the value provided.


6.  videoMaxHeight
Definition of this constraint:

Units for this constraint are:  pixels.
When returned as a capability:  the browser MUST return the maximum height, in pixels, of any resolution the video device is capable of providing.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream whose height is less than or equal to the value provided.


7.  videoMinAspectratio
Definition of this constraint:

Units for this constraint are:  width pixels per height pixel
When returned as a capability:  the browser MUST return the minimum ratio of width pixels to height pixels of any resolution the video device is capable of providing.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream whose ratio of width pixels to height pixels is greater than or equal to the value provided.


8.  videoMaxAspectratio
Definition of this constraint:

Units for this constraint are:  width pixels per height pixel
When returned as a capability:  the browser MUST return the maximum ratio of width pixels to height pixels of any resolution the video device is capable of providing.
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream whose ratio of width pixels to height pixels is less than or equal to the value provided.


9. videoEnumCameraDirection
Definition of this constraint:

Allowed values for this constraint are:  "user", "environment"
When returned as a capability:  the browser MUST return the direction the camera device faces (the user or the environment)
When used as a constraint:  This constraint is satisfied if and only if the browser returns any video stream which faces in the direction given by the value.

Received on Monday, 23 April 2012 18:41:37 UTC