Constraints structure and Capabilities API

Folks,

Cullen, Neil, Tim Panton, and I have come up with the following proposal for Constraints and for Capabilities.  The latter is for the trusted case.  We can work out a subset of information to provide in the case where the Javascript/web developer is not trusted.

I'd like to get any general feedback now before fleshing out the precise wording to go into the spec documents and registry drafts.

Thoughts?

-- dan

*******************************************

Overview
---------
Instead of Hints we have Constraints.  Why?  What's the difference?
The word "Hints" implies that they are suggestions by a web developer that may be completely ignored by the browser.  To address both the needs of implementers and the needs of web developers, there are four properties that the API should have:
1) it should allow the web developer to specify what he would like, IN TERMS THAT ARE MEANINGFUL TO THE BROWSER/IMPLEMENTER
2) it should allow the web developer to specify preferences when he can't get his first choice
3) it should allow the web developer to specify minimal criteria that he will accept
4) it should provide enough information to the browser that it can, on its own, handle many real-world occurrences such as congestion, user-driven screen resizing, etc.

We believe the Constraints structure satisfies these properties.

With regard to the specific list of constraints, the goal was to identify constraints that a) are meaningful to the browser, b) are likely to apply to all media of the given type (video or audio), and c) are likely to be of use to web developers.

Note that we may eventually need to introduce a distinction between PeerConnection capabilities and constraints and getUserMedia capabilities and constraints.  The former is for the actual media that will be communicated between the browsers.  The APIs below were designed with the former in mind but may be similar to what we will want for getUserMedia.

Constraints API/structure
--------
The Constraints API is not really an API, but rather a data structure.  The structure allows a web developer to specify an ordered list of key-value pairs, where each key is a constraint the browser is to attempt to satisfy.  For any two listed constraints, if the browser is able to satisfy one or the other, but not both, the browser MUST satisfy the one that comes earlier in the list. Additionally, each constraint may be marked as mandatory.  If the browser is unable to satisfy all mandatory constraints, the media request call MUST fail with an appropriate error message and list the mandatory constraints it was unable to satisfy.

An initial list of valid constraints is:

video-min-width:  minimum width in pixels
video-max-width:  maximum width in pixels
video-min-height:  minimum height in pixels
video-max-height:  maximum height in pixels
video-min-aspectratio:  minimum width-to-height ratio
video-max-aspectratio:  maximum width-to-height ratio
video-min-framerate:  minimum number of frames per second
video-max-framerate:  maximum number of frames per second
video-min-pixelrate:  minimum pixel transmission rate, in megapixels per second
video-max-pixelrate:  maximum pixel transmission rate, in megapixels per second
video-min-timebetweenkeyframes;  minimum time, in milliseconds, between key/reference frames
video-max-timebetweenkeyframes:  maximum time, in milliseconds, between key/reference frames
video-min-bandwidth:  minimum bandwidth, in megabits per second
video-max-bandwidth:  maximum bandwidth, in megabits per second
video-lowmotion:  whether the coding is to be optimized for scenes where capturing fine detail is more important than motion.  Allowed values are "lowmotion" and "generic".
video-autowhitebalance:  whether Automatic White Balancing is to be turned on.  Allowed values are "on" and "off".
audio-min-bandwidth:  minimum bandwidth, in kilobits per second
audio-max-bandwidth:  maximum bandwidth, in kilobits per second
audio-min-mos:  minimum Mean Opinion Score, ranging from 1 to 5
audio-max-mos:  maximum Mean Opinion Score, ranging from 1 to 5
audio-min-codinglatency:  minimum coding latency, in milliseconds
audio-max-codinglatency:  maximum coding latency, in milliseconds
audio-min-samplingrate:  minimum sampling rate, in samples per second
audio-max-samplingrate:  maximum sampling rate, in samples per second
audio-voiceorgeneric:  whether the coding is to be optimized for voice or for general audio such as music.  Allowed values are "voice" and "generic".
audio-gaincontrol:  whether automatic gain control is to be turned on.  Allowed values are "on" and "off".
audio-echocancellation:  whether echo cancellation is to be turned on.  Allowed values are "on" and "off".

These constraints will be stored in a Constraints IANA registry which can be extended via the Expert Review policy.


Example:
 {0:{video-min-height:600, mandatory:true},
  1:{video-max-aspectratio:1.333333333333},
  2:{video-min-timebetweenrefframes:20},
  3:{video-max-bandwidth:500, mandatory:true},
  4:{video-min-framerate:30},
  5:(video-autowhitebalance:on}}

Capabilties API
---------------
The capabilities API, broadly speaking, is the constraints API, per device/media stream/channel.

A call to getCapabilities() returns a JavaScript Array containing, for each device/media stream/channel, all relevant constraints (the constraints as specified by the constraints API). For example,

{camera001:{
   video-min-width:  800,
   video-max-width:  1024,
   video-min-height:  600,
   video-max-height:  768,
   video-min-aspectratio:  1.333333333333,
   video-max-aspectratio:  1.333333333333,
   video-min-framerate:  24,
   video-max-framerate:  60,
   video-min-pixelrate:  15,
   video-max-pixelrate:  47,
   video-min-timebetweenkeyframes;  20,
   video-max-timebetweenkeyframes:  40,
   video-min-bandwidth:  1.5,
   video-max-bandwidth:  3.5},
 camera002:{
   video-min-width:  1600,
   video-max-width:  1920,
   video-min-height:  1080,
   video-max-height:  1200,
   video-min-aspectratio:  1.33333333333,
   video-max-aspectratio:  1.77777777777,
   video-min-framerate:  24,
   video-max-framerate:  120,
   video-min-pixelrate:  57.6,
   video-max-pixelrate:  248,
   video-min-timebetweenkeyframes;  20,
   video-max-timebetweenkeyframes:  40,
   video-min-bandwidth:  8,
   video-max-bandwidth:  29.4},
 audio001:{
   audio-min-bandwidth:  1.4,
   audio-max-bandwidth:  128,
   audio-min-mos:  2,
   audio-max-mos:  5,
   audio-min-codinglatency:  10,
   audio-max-codinglatency:  50,
   audio-min-samplingrate:  8000,
   audio-max-samplingrate:  48000}}

Note that bandwidth and latency capabilities reflect the combined effect of that device and any codecs we have available to use with it.

Received on Thursday, 23 February 2012 20:29:56 UTC