Moving getUserMedia to best-effort

I've been thinking a lot about Constraints and their current integration into getUserMedia, and the points that Rich made in a previous discussion: http://lists.w3.org/Archives/Public/public-media-capture/2012May/0063.html. This also came up again in a recent DAP teleconference.

I like constraints. However, I'm starting to believe that putting them front-and-center as parameters to getUserMedia is a bad idea. Rich articulates this rather well in his last mail (referenced above), but let me put forth my own view below and then offer an actionable suggestion.

Here's the goal with constraints (as I see it):

*         Provide a set of optional constraints to help the UA (and ultimately end-users) select the best device configuration for the scenario the developer would like.

*         Provide a set of mandatory constraints to restrict the ability to get media (at all) if the web developer's scenario cannot be met.

Here's the potential problem (as I see it):

1.       UAs that run untrusted code (e.g., browsers) won't provide a "getCapabilities" API, so that (at the outset) developers won't know the capabilities that are available for the devices of any given UA. This is good and I support that.

2.       Without knowing the capabilities, developers may set a "quality bar" (hopefully using optional constraints) to try and get the best results for their experience.

3.       Developer may get ornery and use mandatory constraints.

4.       UAs come with all sorts of capabilities and form factors, from mobile to desktop to home-security hardware. Mandatory constraints will cut-off some set of these UAs that don't have capabilities that meet the bar.

5.       These devices/UAs will not like being excluded, and will implement a variation of the official spec that makes mandatory constraints optional (or some other workaround).

Like I said before: I like constraints. But the above problem leads to an exclusionary design, rather than a best-effort design, which has always been a hallmark of the web. Also, constraints without capabilities is generally problematic.

My proposal is simple (in principle): Move constraints from getUserMedia to LocalMediaStream and add a getCapabilities API to LocalMediaStream too. Revert getUserMedia to its former best-effort approach (with respect to the QoS of either the requested video or audio tracks) and use the fact that permission will already have been granted to a particular combination of tracks (by the time a LocalMediaStream is available) as implicit permission to get the capabilities of the devices furnishing the existing tracks.

The previous problem steps now work out as follows:


a.       UAs that run untrusted code (e.g., browsers) will provide a "getCapabilities" API, but only after granting permission for those capabilities to be released along with the media they are providing.

b.      Developers can negotiate a level of quality that is applicable and relevant to the existing capabilities of the device to try and get the best results for their experience.

c.       Developer may get ornery and reject the media stream if it doesn't meet some quality bar (but this has always been possible with the stop() API) and is less likely under these circumstances.

Since operations that obtain the capabilities and assign the constraints will be interacting with the local media resources that are furnishing the initial MediaStreamTrack(s) of a LocalMediaStream, and given that LocalMediaStream objects are strongly tied to local media resources (even though their track lists are mutable over time), I propose the following:

interface LocalMediaStream : MediaStream {
    void stop ();
    // proposal ******************************
    MediaStreamConstraints? getUserMediaCapabilities();
    void applyConstraints(MediaStreamConstraints? constraints);
};

"getUserMediaCapabilities" will, of course, only report the capabilities for all the local media resources that have been granted permission previously by getUserMedia.

"applyConstraints" would only work against the local media resources that have been granted permission previously by getUserMedia. Additionally, this API may have the side-effect of stopping any affected existing MediaStreamTrack objects and creating new MediaStreamTrack objects (if any) that now conform to the constraints. The new tracks could be automatically added to the LocalMediaStream (as if videoTracks/audioTracks.add had been called) or furnished in a callback/eventhandler.

We could also add events to help the developer understand what happened when the constraints were applied (e.g., for error conditions).