Re: Moving getUserMedia to best-effort

On 07/27/2012 04:18 PM, Travis Leithead wrote:
>
> I've been thinking a lot about Constraints and their current 
> integration into getUserMedia, and the points that Rich made in a 
> previous discussion: 
> http://lists.w3.org/Archives/Public/public-media-capture/2012May/0063.html. 
> This also came up again in a recent DAP teleconference.
>
> I like constraints. However, I'm starting to believe that putting them 
> front-and-center as parameters to getUserMedia is a bad idea. Rich 
> articulates this rather well in his last mail (referenced above), but 
> let me put forth my own view below and then offer an actionable 
> suggestion.
>
> Here's the goal with constraints (as I see it):
>
> ·Provide a set of optional constraints to help the UA (and ultimately 
> end-users) select the best device configuration for the scenario the 
> developer would like.
>
> ·Provide a set of mandatory constraints to restrict the ability to get 
> media (at all) if the web developer's scenario cannot be met.
>
> Here's the potential problem (as I see it):
>
> 1.UAs that run untrusted code (e.g., browsers) won't provide a 
> "getCapabilities" API, so that (at the outset) developers won't know 
> the capabilities that are available for the devices of any given UA. 
> This is good and I support that.
>
> 2.Without knowing the capabilities, developers may set a "quality bar" 
> (hopefully using optional constraints) to try and get the best results 
> for their experience.
>
> 3.Developer may get ornery and use mandatory constraints.
>
> 4.UAs come with all sorts of capabilities and form factors, from 
> mobile to desktop to home-security hardware. Mandatory constraints 
> /will /cut-off some set of these UAs that don't have capabilities that 
> meet the bar.
>
> 5.These devices/UAs will not like being excluded, and will implement a 
> variation of the official spec that makes mandatory constraints 
> optional (or some other workaround).
>
> Like I said before: I like constraints. But the above problem leads to 
> an exclusionary design, rather than a best-effort design, which has 
> always been a hallmark of the web. Also, constraints without 
> capabilities is generally problematic.
>
I don't see that logic as compelling.
A competing logic is:

- Deveopers use mandatory constraints exactly in order to exclude some 
devices, because they know that users on those devices will have a bad 
experience
- Some UA implementors will design non-conformant UAs in order to run on 
those devices
- Some users will have a bad experience

Why should we make it hard to decide who to blame in this situation?

> My proposal is simple (in principle): Move constraints from 
> getUserMedia to LocalMediaStream
>

This would mean that you can't apply constraints up front, you can only 
apply constraints after getting access to the device. Why is that an 
advantage?

> and add a getCapabilities API to LocalMediaStream too. Revert 
> getUserMedia to its former best-effort approach (with respect to the 
> QoS of either the requested video or audio tracks) and use the fact 
> that permission will already have been granted to a particular 
> combination of tracks (by the time a LocalMediaStream is available) as 
> implicit permission to get the capabilities of the devices furnishing 
> the existing tracks.
>
You mean that you want the user to be asked for permission on devices 
that the application will then reject as useless. Why is that a good UI 
design?
>
> The previous problem steps now work out as follows:
>
> a.UAs that run untrusted code (e.g., browsers) will provide a 
> "getCapabilities" API, but only /after/ granting permission for those 
> capabilities to be released along with the media they are providing.
>
> b.Developers can negotiate a level of quality that is applicable and 
> relevant to the existing capabilities of the device to try and get the 
> best results for their experience.
>
> c.Developer may get ornery and reject the media stream if it doesn't 
> meet some quality bar (but this has always been possible with the 
> stop() API) and is less likely under these circumstances.
>
> Since operations that obtain the capabilities and assign the 
> constraints will be interacting with the local media resources that 
> are furnishing the initial MediaStreamTrack(s) of a LocalMediaStream, 
> and given that LocalMediaStream objects are strongly tied to local 
> media resources (even though their track lists are mutable over time), 
> I propose the following:
>
> interface LocalMediaStream : MediaStream {
>
>     void stop ();
>
>     // proposal ******************************
>
>     MediaStreamConstraints? getUserMediaCapabilities();
>
>     void applyConstraints(MediaStreamConstraints? constraints);
>
> };
>
> "getUserMediaCapabilities" will, of course, /only/ report the 
> capabilities for all the local media resources that have been granted 
> permission previously by getUserMedia.
>
> "applyConstraints" would only work against the local media resources 
> that have been granted permission previously by getUserMedia. 
> Additionally, this API may have the side-effect of stopping any 
> affected existing MediaStreamTrack objects and creating new 
> MediaStreamTrack objects (if any) that now conform to the constraints. 
> The new tracks could be automatically added to the LocalMediaStream 
> (as if videoTracks/audioTracks.add had been called) or furnished in a 
> callback/eventhandler.
>
> We could also add events to help the developer understand what 
> happened when the constraints were applied (e.g., for error conditions).
>

I actually see this API as making a lot of sense for *changing* media 
capabilities on a stream (which may succeed or may fail, so you need a 
return value or callback from applyConstraint).

But the only difference I see between
getUserMedia(function(flow) {
   if (flow.applyConstraints(constraints)) {
      handleSuccess()
   } else
      handleFailure()
    }
}

and

flow = getUserMedia(constraints, handleSuccess, handleFailure)

is that you enforce that the user has to be prompted for permission to 
the device before rejecting it.

I don't see the advantage in that.

                     Harald