Re: Revised Constraints modification API proposal

Thanks very much for this, Travis!

One of the aspects of your proposal that I like is the additional power it gives to constrain and control at the device level once specific tracks have been obtained from specific devices.  It provides this through very clear controls on *local* tracks.

Other comments in-line.

-- dan

On Aug 21, 2012, at 4:56 PM, Travis Leithead wrote:

> Hi folks! 
> 
> In preparation for the upcoming Telco, I've adjusted the "Constraints 
> modification API" proposal, originally proposed in [1] based on feedback 
> from Harald and others, as well as incorporating ideas from Rich's recent 
> proposal [2]. I hope that this proposal furthers the convergence that we
> all seek. I took the liberty of writing this out in narrative form, so 
> please read through to the end to get the full picture of this proposal.
> 
> The rest of this proposal is broken into three sections:
> 1. Creating the "suitable" home for the APIs in question
> 2. Defining how to get capabilities
> 3. Defining how to apply constraints
> 
> As a reminder, the goal of this proposal is to facilitate "informed 
> constraints" (i.e., allow constraints to be applied after existing client 
> capabilities are known) in order to avoid potential pitfalls of blindly 
> over-constrained use of getUserMedia across a range of different devices.

I always wanted to have capabilities along with constraints, and I'm pleased to see (finally) a realistic capabilities proposal.  Given that we will not always have capabilities available for privacy reasons, I'd like to understand better these "potential pitfalls of blindly over-constrained use".  I have heard that mentioned but have not yet seen enough evidence to convince me that this is a real problem.  Could you point me to some examples?

> 
> A secondary goal is to provide the right set of APIs for uniformly working
> with the devices that supply the local media stream tracks, for future APIs
> and scenarios we may wish to add.
> 
> If this proposal is adopted, I would expect that the existing constraint
> usage in getUserMedia could be significantly scaled back, if not removed
> altogether.

With the device-level only focus below, I don't think that would quite work.  One of the goals of the constraints approach was to free a developer who just wants "a camera" from having to worry about which one.  They just specify their constraints, mandatory or optional based on their needs, and if they get something it is something they can use.

> 
> Thanks!
> 
> -------------
> 
> 1. Create a suitable home for constraint application and capability 
>   retrieval that is strongly tied to the track concept.
> 
> In order to define a capabilities/constraints API that supports modifications,
> we need to carefully consider where it should live. I believe that it needs 
> to exist only for "LocalMediaStreams" so that there is no confusion/ambiguity 
> surrounding cloned MediaStreams or MediaStream objects obtained from a 
> PeerConnection (remotely over a network). 
> Note that providing a signaling channel or any other notification framework
> for constraint changes or capability requests over the network is out of scope 
> for this proposal.
> 
> The existing tracks concept seems like a good place for this API. However,
> because tracks can be added/removed from various MediaStreams (and 
> LocalMediaStreams), there's no real guarantee that the tracks you're dealing
> with in the LocalMediaStream are the ones that are related to your "local" 
> device. It's also confusing for developers if some tracks have special "local"
> capabilities while others don't (in the same track list). This is a problem 
> that needs a solution if a consistent and reliable method for interacting with 
> just your "local" (approved) tracks is to be designed. I'll call this the "transient-
> track" problem.
> 
> Note: I'm not totally satisfied with this proposal, because even LocalMediaStream
> objects are potentially transient, and if the developer loses the reference to the
> LocalMediaStream, then they're out of luck. It's also a little weird to have two 
> instances of LocalMediaStreams (from two calls to getUserMedia), but have basically
> the same information provided in parallel between them. Perhaps that's not a bit deal.
> Feedback, as always, is welcome.
> 
> A new LocalMediaStreamTrackList is defined in order to address the transient-track 
> problem noted above. These track lists are different from MediaStreamTrackLists in 
> one notable way:
> 
> * There is no ability to remove/add tracks to these lists. The addition and removal of
> tracks in these lists are managed exclusively by the user agent.
> 
> The new track list's purpose is to aggregate all the "local" media stream tracks
> together for enumeration, settings, and constraint application. It also serves
> as a convenient list of the "active" local devices that have been approved by the
> user.
> 
> // +++ New interface
> interface LocalMediaStreamTrackList {
>   readonly attribute unsigned long length;
>   getter MediaStreamTrack (unsigned long index);
>   attribute EventHandler onaddtrack;
>   attribute EventHandler onremovetrack;
> };
> 
> The developer can be notified when new local audio/video devices are enabled
> by the user (by subsequent calls to getUserMedia for example) or when local audio/
> video devices have stopped by registering for the [familiar] onaddtrack/onremovetrack
> event handlers.
> 
> There are not add/remove APIs, as the management of this list is done exclusively 
> by the user agent as previously noted.
> 
> The LocalMediaStreamTrackList is surfaced on the LocalMediaStream interface as two
> new properties:
> 
> // Existing definition...
> interface LocalMediaStream: MediaStream {
>   // Existing stop API
>   void stop ();
>   // +++ New
>   readonly attribute LocalMediaStreamTrackList audioDevices;
>   readonly attribute LocalMediaStreamTrackList videoDevices;
> };
> 
> Each audio/video LocalMediaStreamTrackList is kept up-to-date among all 
> LocalMediaStream instances, so that it doesn't matter which instance of a
> LocalMediaStream object is used, it will always have the aggregate information
> for all active local audio/video devices [previously] approved by the user.
> 
> These two lists now represent (conceptually) the set of active devices
> supplying media streams on the user's local box. Each track within these 
> lists essentially represents a local device. So these lists are an enumeration
> of the users active approved devices (not necessarily all the devices that
> the user has available). Since these are approved devices, it follows that 
> requesting capabilities of and applying constraints to these devices is 
> acceptable without requesting additional permissions from the user.
> 
> Note that my proposal as-is doesn't describe how the developer can discover
> that there are more devices available.
> 
> 2. Get the capabilities in terms of Track objects.
> 
> In the proposal above, notice that the LocalMediaStreamTrackList contains a 
> list of MediaStreamTrack objects. As already proposed by Rich in [2], I also 
> recommend a factoring of MediaStreamTrack into two local derived types: 
> LocalVideoStreamTrack and LocalAudioStreamTrack (my names include the prefix 
> "local" for clarity):
> 
> // +++ new factored interface for video-specific APIs
> interface LocalVideoStreamTrack : MediaStreamTrack {
> }
> 
> // +++ new factored interface for audio-specific APIs
> interface LocalAudioStreamTrack : MediaStreamTrack {
> }
> 
> The most-derived interfaces are always returned from the LocalMediaStreamTrackList.
> 
> The current state of the track can be reflected in APIs added to these objects.
> As described in [2], for the LocalVideoStreamTrack these might include:
> * autoFocusMode
> * currentZoom
> * currentFlashMode
> * currentDisplayOrientation
> * viewingAngle
> * etc.
> LocalAudioStreamTracks might have:
> * currentAudioLevel
> 
> Each local track contains the "current" settings, which can be directly 
> inspected. In order to find the "range" of the available settings, 
> e.g., the capabilities of a given track, I propose a common API across 
> both track types:
> 
> typedef sequence<MediaTrackConstraint>? LocalDeviceCapabilites;
> LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
> 
> This API is located on the *track list* (i.e., LocalMediaStreamTrackList) 
> in order to be able to operate on either a single track (e.g., "give me the 
> capabilities for a single device") or all the available devices (e.g., "give
> me the capabilities for all the available devices"). (By "available", I mean
> those devices already approved by getUserMedia).
> 
> // New interface
> interface LocalMediaStreamTrackList {
>   readonly attribute unsigned long length;
>   getter MediaStreamTrack (unsigned long index);
>   attribute EventHandler onaddtrack;
>   attribute EventHandler onremovetrack;
>   // +++ capabilities API
>   LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
> };
> 
> If no parameter is provided, then the combined capabilities of all available tracks 
> for the given list (audio/video) is returned. If the developer only has a single track,
> the API is very simple:
> 
> var caps = localStream.videoDevices.getCapabilities();
> 
> If there are multiple video devices currently active, then the developer 
> uses the same code as above to get all the combined capabilities, or they can
> pick-and-choose:
> 
> // Get the capabilities for the second active video device
> var caps = localStream.videoDevices.getCapabilities(1);
> 
> The capabilities returned are the set of capability "ranges" that the device supports, 
> suitable for use in the constraints API.
> 
> 3. Apply constraints to Track objects
> 
> Now that the capabilities can be inspected for approved devices (all or per-track),
> constraints can be applied to them. As with getting the capabilities, constraint 
> application is applied either directly to a MediaStreamTrack or to all currently 
> active local tracks in a given list (audio/video).
> 
> // New interface
> interface LocalMediaStreamTrackList {
>   readonly attribute unsigned long length;
>   getter MediaStreamTrack (unsigned long index);
>   attribute EventHandler onaddtrack;
>   attribute EventHandler onremovetrack;
>   LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
>   // +++ Constraints API
>   void applySettings(MediaTrackConstraints, optional (MediaStreamTrack or unsigned long) track);
> };
> 
> I'm not a believer in track object mutation. In other words, I don't believe that
> a track should be able to be "in-place" updated by applying a constraint to it. 
> Rather, my view is that if there is a change that the device makes in response
> to the application of new constraints, old tracks are stopped and new track objects
> are created in response. This makes the application of constraints implicitly 
> asynchronous, which prevents developers from taking a dependency on the "immediate
> application" of constraints to a given track which may not be possible for all devices.
> 
> The "applySettings" API (I renamed it to sound more friendly), acts on all the local 
> media tracks in the list by default, or targets only a specific track if one is 
> indicated in the 2nd parameter.
> 
> If targeted toward a specific track, the new constraints are evaluated against 
> the current track settings, and if the current settings fall within the new 
> constraints then no change is made. If the constraints affect the current settings 
> of the track, then that track is stopped (and consequently removed from the
> LocalMediaStreamTrackList as a result) and if the device associated with that track
> can support the new constraints, then a new track is created and added to the
> LocalMediaStreamTrackList. (All these actions trigger the appropriate add/remove 
> events.)

This isn't necessary.  Since getUserMedia() still exists, a developer can can always call that to get completely new tracks.
If I tweak the constraints of an existing track I would expect that track to be modified (if it satisfies the constraints).

> 
> If not targeted toward a specific track (applying to all the current active devices),
> the above algorithm is run for every track in the list.
> 
> To summarize:
> This proposal adds two new track lists to local media streams which correspond to
> the "active" local devices supplying the tracks which have been approved by
> getUserMedia. The proposal incorporates a getCapabilities and applySettings API onto
> these tracks lists which can operate on the list as a whole or on individual tracks
> in the list. This proposal also recommends that individual local tracks be augmented
> with device-specific "current" settings, though the details of these settings were 
> only alluded to via another proposal [2].
> 
> [1] http://lists.w3.org/Archives/Public/public-media-capture/2012Jul/0069.html
> [2] http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0032.html
> 
>

Received on Wednesday, 22 August 2012 18:02:05 UTC