Revised Constraints modification API proposal

Hi folks! 

In preparation for the upcoming Telco, I've adjusted the "Constraints 
modification API" proposal, originally proposed in [1] based on feedback 
from Harald and others, as well as incorporating ideas from Rich's recent 
proposal [2]. I hope that this proposal furthers the convergence that we
all seek. I took the liberty of writing this out in narrative form, so 
please read through to the end to get the full picture of this proposal.

The rest of this proposal is broken into three sections:
1. Creating the "suitable" home for the APIs in question
2. Defining how to get capabilities
3. Defining how to apply constraints

As a reminder, the goal of this proposal is to facilitate "informed 
constraints" (i.e., allow constraints to be applied after existing client 
capabilities are known) in order to avoid potential pitfalls of blindly 
over-constrained use of getUserMedia across a range of different devices.

A secondary goal is to provide the right set of APIs for uniformly working
with the devices that supply the local media stream tracks, for future APIs
and scenarios we may wish to add.

If this proposal is adopted, I would expect that the existing constraint
usage in getUserMedia could be significantly scaled back, if not removed
altogether.

Thanks!

-------------

1. Create a suitable home for constraint application and capability 
   retrieval that is strongly tied to the track concept.

In order to define a capabilities/constraints API that supports modifications,
we need to carefully consider where it should live. I believe that it needs 
to exist only for "LocalMediaStreams" so that there is no confusion/ambiguity 
surrounding cloned MediaStreams or MediaStream objects obtained from a 
PeerConnection (remotely over a network). 
Note that providing a signaling channel or any other notification framework
for constraint changes or capability requests over the network is out of scope 
for this proposal.

The existing tracks concept seems like a good place for this API. However,
because tracks can be added/removed from various MediaStreams (and 
LocalMediaStreams), there's no real guarantee that the tracks you're dealing
with in the LocalMediaStream are the ones that are related to your "local" 
device. It's also confusing for developers if some tracks have special "local"
capabilities while others don't (in the same track list). This is a problem 
that needs a solution if a consistent and reliable method for interacting with 
just your "local" (approved) tracks is to be designed. I'll call this the "transient-
track" problem.

Note: I'm not totally satisfied with this proposal, because even LocalMediaStream
objects are potentially transient, and if the developer loses the reference to the
LocalMediaStream, then they're out of luck. It's also a little weird to have two 
instances of LocalMediaStreams (from two calls to getUserMedia), but have basically
the same information provided in parallel between them. Perhaps that's not a bit deal.
Feedback, as always, is welcome.

A new LocalMediaStreamTrackList is defined in order to address the transient-track 
problem noted above. These track lists are different from MediaStreamTrackLists in 
one notable way:

* There is no ability to remove/add tracks to these lists. The addition and removal of
tracks in these lists are managed exclusively by the user agent.

The new track list's purpose is to aggregate all the "local" media stream tracks
together for enumeration, settings, and constraint application. It also serves
as a convenient list of the "active" local devices that have been approved by the
user.

// +++ New interface
interface LocalMediaStreamTrackList {
   readonly attribute unsigned long length;
   getter MediaStreamTrack (unsigned long index);
   attribute EventHandler onaddtrack;
   attribute EventHandler onremovetrack;
};

The developer can be notified when new local audio/video devices are enabled
by the user (by subsequent calls to getUserMedia for example) or when local audio/
video devices have stopped by registering for the [familiar] onaddtrack/onremovetrack
event handlers.

There are not add/remove APIs, as the management of this list is done exclusively 
by the user agent as previously noted.

The LocalMediaStreamTrackList is surfaced on the LocalMediaStream interface as two
new properties:

// Existing definition...
interface LocalMediaStream: MediaStream {
   // Existing stop API
   void stop ();
   // +++ New
   readonly attribute LocalMediaStreamTrackList audioDevices;
   readonly attribute LocalMediaStreamTrackList videoDevices;
};

Each audio/video LocalMediaStreamTrackList is kept up-to-date among all 
LocalMediaStream instances, so that it doesn't matter which instance of a
LocalMediaStream object is used, it will always have the aggregate information
for all active local audio/video devices [previously] approved by the user.

These two lists now represent (conceptually) the set of active devices
supplying media streams on the user's local box. Each track within these 
lists essentially represents a local device. So these lists are an enumeration
of the users active approved devices (not necessarily all the devices that
the user has available). Since these are approved devices, it follows that 
requesting capabilities of and applying constraints to these devices is 
acceptable without requesting additional permissions from the user.

Note that my proposal as-is doesn't describe how the developer can discover
that there are more devices available.

2. Get the capabilities in terms of Track objects.

In the proposal above, notice that the LocalMediaStreamTrackList contains a 
list of MediaStreamTrack objects. As already proposed by Rich in [2], I also 
recommend a factoring of MediaStreamTrack into two local derived types: 
LocalVideoStreamTrack and LocalAudioStreamTrack (my names include the prefix 
"local" for clarity):

// +++ new factored interface for video-specific APIs
interface LocalVideoStreamTrack : MediaStreamTrack {
}

// +++ new factored interface for audio-specific APIs
interface LocalAudioStreamTrack : MediaStreamTrack {
}

The most-derived interfaces are always returned from the LocalMediaStreamTrackList.

The current state of the track can be reflected in APIs added to these objects.
As described in [2], for the LocalVideoStreamTrack these might include:
* autoFocusMode
* currentZoom
* currentFlashMode
* currentDisplayOrientation
* viewingAngle
* etc.
LocalAudioStreamTracks might have:
* currentAudioLevel

Each local track contains the "current" settings, which can be directly 
inspected. In order to find the "range" of the available settings, 
e.g., the capabilities of a given track, I propose a common API across 
both track types:

typedef sequence<MediaTrackConstraint>? LocalDeviceCapabilites;
LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);

This API is located on the *track list* (i.e., LocalMediaStreamTrackList) 
in order to be able to operate on either a single track (e.g., "give me the 
capabilities for a single device") or all the available devices (e.g., "give
me the capabilities for all the available devices"). (By "available", I mean
those devices already approved by getUserMedia).

// New interface
interface LocalMediaStreamTrackList {
   readonly attribute unsigned long length;
   getter MediaStreamTrack (unsigned long index);
   attribute EventHandler onaddtrack;
   attribute EventHandler onremovetrack;
   // +++ capabilities API
   LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
};

If no parameter is provided, then the combined capabilities of all available tracks 
for the given list (audio/video) is returned. If the developer only has a single track,
the API is very simple:

var caps = localStream.videoDevices.getCapabilities();

If there are multiple video devices currently active, then the developer 
uses the same code as above to get all the combined capabilities, or they can
pick-and-choose:

// Get the capabilities for the second active video device
var caps = localStream.videoDevices.getCapabilities(1);

The capabilities returned are the set of capability "ranges" that the device supports, 
suitable for use in the constraints API.

3. Apply constraints to Track objects

Now that the capabilities can be inspected for approved devices (all or per-track),
constraints can be applied to them. As with getting the capabilities, constraint 
application is applied either directly to a MediaStreamTrack or to all currently 
active local tracks in a given list (audio/video).

// New interface
interface LocalMediaStreamTrackList {
   readonly attribute unsigned long length;
   getter MediaStreamTrack (unsigned long index);
   attribute EventHandler onaddtrack;
   attribute EventHandler onremovetrack;
   LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
   // +++ Constraints API
   void applySettings(MediaTrackConstraints, optional (MediaStreamTrack or unsigned long) track);
};

I'm not a believer in track object mutation. In other words, I don't believe that
a track should be able to be "in-place" updated by applying a constraint to it. 
Rather, my view is that if there is a change that the device makes in response
to the application of new constraints, old tracks are stopped and new track objects
are created in response. This makes the application of constraints implicitly 
asynchronous, which prevents developers from taking a dependency on the "immediate
application" of constraints to a given track which may not be possible for all devices.

The "applySettings" API (I renamed it to sound more friendly), acts on all the local 
media tracks in the list by default, or targets only a specific track if one is 
indicated in the 2nd parameter.

If targeted toward a specific track, the new constraints are evaluated against 
the current track settings, and if the current settings fall within the new 
constraints then no change is made. If the constraints affect the current settings 
of the track, then that track is stopped (and consequently removed from the
LocalMediaStreamTrackList as a result) and if the device associated with that track
can support the new constraints, then a new track is created and added to the
LocalMediaStreamTrackList. (All these actions trigger the appropriate add/remove 
events.)

If not targeted toward a specific track (applying to all the current active devices),
the above algorithm is run for every track in the list.

To summarize:
This proposal adds two new track lists to local media streams which correspond to
the "active" local devices supplying the tracks which have been approved by
getUserMedia. The proposal incorporates a getCapabilities and applySettings API onto
these tracks lists which can operate on the list as a whole or on individual tracks
in the list. This proposal also recommends that individual local tracks be augmented
with device-specific "current" settings, though the details of these settings were 
only alluded to via another proposal [2].

[1] http://lists.w3.org/Archives/Public/public-media-capture/2012Jul/0069.html
[2] http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0032.html

Received on Tuesday, 21 August 2012 20:57:32 UTC