Re: Revised Constraints modification API proposal

On 08/21/2012 10:56 PM, Travis Leithead wrote:
> Hi folks!
>
> In preparation for the upcoming Telco, I've adjusted the "Constraints
> modification API" proposal, originally proposed in [1] based on feedback
> from Harald and others, as well as incorporating ideas from Rich's recent
> proposal [2]. I hope that this proposal furthers the convergence that we
> all seek. I took the liberty of writing this out in narrative form, so
> please read through to the end to get the full picture of this proposal.
>
> The rest of this proposal is broken into three sections:
> 1. Creating the "suitable" home for the APIs in question
> 2. Defining how to get capabilities
> 3. Defining how to apply constraints
>
> As a reminder, the goal of this proposal is to facilitate "informed
> constraints" (i.e., allow constraints to be applied after existing client
> capabilities are known) in order to avoid potential pitfalls of blindly
> over-constrained use of getUserMedia across a range of different devices.
>
> A secondary goal is to provide the right set of APIs for uniformly working
> with the devices that supply the local media stream tracks, for future APIs
> and scenarios we may wish to add.
>
> If this proposal is adopted, I would expect that the existing constraint
> usage in getUserMedia could be significantly scaled back, if not removed
> altogether.
>
> Thanks!
>
> -------------
>
> 1. Create a suitable home for constraint application and capability
>     retrieval that is strongly tied to the track concept.
>
> In order to define a capabilities/constraints API that supports modifications,
> we need to carefully consider where it should live. I believe that it needs
> to exist only for "LocalMediaStreams" so that there is no confusion/ambiguity
> surrounding cloned MediaStreams or MediaStream objects obtained from a
> PeerConnection (remotely over a network).
I fell off the logic chain right here.

I believe there's a need to specify and modify constraints on media 
streams that are not obtained from GetUserMedia, so I believe the API 
needs to exist as a MediaStreamTrack API, not a LocalMediaStreamTrack API.
However, the constraints that make sense may be different for the two cases.

> Note that providing a signaling channel or any other notification framework
> for constraint changes or capability requests over the network is out of scope
> for this proposal.
>
> The existing tracks concept seems like a good place for this API. However,
> because tracks can be added/removed from various MediaStreams (and
> LocalMediaStreams), there's no real guarantee that the tracks you're dealing
> with in the LocalMediaStream are the ones that are related to your "local"
> device. It's also confusing for developers if some tracks have special "local"
> capabilities while others don't (in the same track list). This is a problem
> that needs a solution if a consistent and reliable method for interacting with
> just your "local" (approved) tracks is to be designed. I'll call this the "transient-
> track" problem.
>
> Note: I'm not totally satisfied with this proposal, because even LocalMediaStream
> objects are potentially transient, and if the developer loses the reference to the
> LocalMediaStream, then they're out of luck.
Why is this a problem? If a developer loses the reference to a 
LocalMediaStream, we have to assume that he's got no further interest in 
modifying it, don't we?
>   It's also a little weird to have two
> instances of LocalMediaStreams (from two calls to getUserMedia), but have basically
> the same information provided in parallel between them. Perhaps that's not a bit deal.
> Feedback, as always, is welcome.
>
> A new LocalMediaStreamTrackList is defined in order to address the transient-track
> problem noted above. These track lists are different from MediaStreamTrackLists in
> one notable way:
>
> * There is no ability to remove/add tracks to these lists. The addition and removal of
> tracks in these lists are managed exclusively by the user agent.
>
> The new track list's purpose is to aggregate all the "local" media stream tracks
> together for enumeration, settings, and constraint application. It also serves
> as a convenient list of the "active" local devices that have been approved by the
> user.
>
> // +++ New interface
> interface LocalMediaStreamTrackList {
>     readonly attribute unsigned long length;
>     getter MediaStreamTrack (unsigned long index);
>     attribute EventHandler onaddtrack;
>     attribute EventHandler onremovetrack;
> };
Is this object a process-wide singleton object? How do applications 
obtain a refererence to this object?

>
> The developer can be notified when new local audio/video devices are enabled
> by the user (by subsequent calls to getUserMedia for example) or when local audio/
> video devices have stopped by registering for the [familiar] onaddtrack/onremovetrack
> event handlers.
>
> There are not add/remove APIs, as the management of this list is done exclusively
> by the user agent as previously noted.
>
> The LocalMediaStreamTrackList is surfaced on the LocalMediaStream interface as two
> new properties:
>
> // Existing definition...
> interface LocalMediaStream: MediaStream {
>     // Existing stop API
>     void stop ();
>     // +++ New
>     readonly attribute LocalMediaStreamTrackList audioDevices;
>     readonly attribute LocalMediaStreamTrackList videoDevices;
> };
>
> Each audio/video LocalMediaStreamTrackList is kept up-to-date among all
> LocalMediaStream instances, so that it doesn't matter which instance of a
> LocalMediaStream object is used, it will always have the aggregate information
> for all active local audio/video devices [previously] approved by the user.
>
> These two lists now represent (conceptually) the set of active devices
> supplying media streams on the user's local box. Each track within these
> lists essentially represents a local device. So these lists are an enumeration
> of the users active approved devices (not necessarily all the devices that
> the user has available). Since these are approved devices, it follows that
> requesting capabilities of and applying constraints to these devices is
> acceptable without requesting additional permissions from the user.
>
> Note that my proposal as-is doesn't describe how the developer can discover
> that there are more devices available.
>
> 2. Get the capabilities in terms of Track objects.
>
> In the proposal above, notice that the LocalMediaStreamTrackList contains a
> list of MediaStreamTrack objects. As already proposed by Rich in [2], I also
> recommend a factoring of MediaStreamTrack into two local derived types:
> LocalVideoStreamTrack and LocalAudioStreamTrack (my names include the prefix
> "local" for clarity):
>
> // +++ new factored interface for video-specific APIs
> interface LocalVideoStreamTrack : MediaStreamTrack {
> }
>
> // +++ new factored interface for audio-specific APIs
> interface LocalAudioStreamTrack : MediaStreamTrack {
> }
>
> The most-derived interfaces are always returned from the LocalMediaStreamTrackList.
>
> The current state of the track can be reflected in APIs added to these objects.
> As described in [2], for the LocalVideoStreamTrack these might include:
> * autoFocusMode
> * currentZoom
> * currentFlashMode
> * currentDisplayOrientation
> * viewingAngle
> * etc.
> LocalAudioStreamTracks might have:
> * currentAudioLevel
>
> Each local track contains the "current" settings, which can be directly
> inspected. In order to find the "range" of the available settings,
> e.g., the capabilities of a given track, I propose a common API across
> both track types:
>
> typedef sequence<MediaTrackConstraint>? LocalDeviceCapabilites;
> LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
>
> This API is located on the *track list* (i.e., LocalMediaStreamTrackList)
> in order to be able to operate on either a single track (e.g., "give me the
> capabilities for a single device") or all the available devices (e.g., "give
> me the capabilities for all the available devices"). (By "available", I mean
> those devices already approved by getUserMedia).
>
> // New interface
> interface LocalMediaStreamTrackList {
>     readonly attribute unsigned long length;
>     getter MediaStreamTrack (unsigned long index);
>     attribute EventHandler onaddtrack;
>     attribute EventHandler onremovetrack;
>     // +++ capabilities API
>     LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
> };
>
> If no parameter is provided, then the combined capabilities of all available tracks
> for the given list (audio/video) is returned. If the developer only has a single track,
> the API is very simple:
>
> var caps = localStream.videoDevices.getCapabilities();
>
> If there are multiple video devices currently active, then the developer
> uses the same code as above to get all the combined capabilities, or they can
> pick-and-choose:
>
> // Get the capabilities for the second active video device
> var caps = localStream.videoDevices.getCapabilities(1);
>
> The capabilities returned are the set of capability "ranges" that the device supports,
> suitable for use in the constraints API.
>
> 3. Apply constraints to Track objects
>
> Now that the capabilities can be inspected for approved devices (all or per-track),
> constraints can be applied to them. As with getting the capabilities, constraint
> application is applied either directly to a MediaStreamTrack or to all currently
> active local tracks in a given list (audio/video).
>
> // New interface
> interface LocalMediaStreamTrackList {
>     readonly attribute unsigned long length;
>     getter MediaStreamTrack (unsigned long index);
>     attribute EventHandler onaddtrack;
>     attribute EventHandler onremovetrack;
>     LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
>     // +++ Constraints API
>     void applySettings(MediaTrackConstraints, optional (MediaStreamTrack or unsigned long) track);
> };
>
> I'm not a believer in track object mutation. In other words, I don't believe that
> a track should be able to be "in-place" updated by applying a constraint to it.
> Rather, my view is that if there is a change that the device makes in response
> to the application of new constraints, old tracks are stopped and new track objects
> are created in response. This makes the application of constraints implicitly
> asynchronous, which prevents developers from taking a dependency on the "immediate
> application" of constraints to a given track which may not be possible for all devices.
I think I'm not getting this.
What happens to references to the old track in other MediaStreams when 
such a replacement happens? Do they silently become invalid, or does 
some other magic happen?
>
> The "applySettings" API (I renamed it to sound more friendly), acts on all the local
> media tracks in the list by default, or targets only a specific track if one is
> indicated in the 2nd parameter.
>
> If targeted toward a specific track, the new constraints are evaluated against
> the current track settings, and if the current settings fall within the new
> constraints then no change is made. If the constraints affect the current settings
> of the track, then that track is stopped (and consequently removed from the
> LocalMediaStreamTrackList as a result) and if the device associated with that track
> can support the new constraints, then a new track is created and added to the
> LocalMediaStreamTrackList. (All these actions trigger the appropriate add/remove
> events.)
>
> If not targeted toward a specific track (applying to all the current active devices),
> the above algorithm is run for every track in the list.
>
> To summarize:
> This proposal adds two new track lists to local media streams which correspond to
> the "active" local devices supplying the tracks which have been approved by
> getUserMedia. The proposal incorporates a getCapabilities and applySettings API onto
> these tracks lists which can operate on the list as a whole or on individual tracks
> in the list. This proposal also recommends that individual local tracks be augmented
> with device-specific "current" settings, though the details of these settings were
> only alluded to via another proposal [2].
>
> [1] http://lists.w3.org/Archives/Public/public-media-capture/2012Jul/0069.html
> [2] http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0032.html
>
>

Received on Wednesday, 22 August 2012 11:04:30 UTC