Re: Revised Constraints modification API proposal

On 2012-08-21 22:56, Travis Leithead wrote:
> Hi folks!
>
> In preparation for the upcoming Telco, I've adjusted the "Constraints
> modification API" proposal, originally proposed in [1] based on feedback
> from Harald and others, as well as incorporating ideas from Rich's recent
> proposal [2]. I hope that this proposal furthers the convergence that we
> all seek. I took the liberty of writing this out in narrative form, so
> please read through to the end to get the full picture of this proposal.
>
> The rest of this proposal is broken into three sections:
> 1. Creating the "suitable" home for the APIs in question
> 2. Defining how to get capabilities
> 3. Defining how to apply constraints
>
> As a reminder, the goal of this proposal is to facilitate "informed
> constraints" (i.e., allow constraints to be applied after existing client
> capabilities are known) in order to avoid potential pitfalls of blindly
> over-constrained use of getUserMedia across a range of different devices.
>
> A secondary goal is to provide the right set of APIs for uniformly working
> with the devices that supply the local media stream tracks, for future APIs
> and scenarios we may wish to add.
>
> If this proposal is adopted, I would expect that the existing constraint
> usage in getUserMedia could be significantly scaled back, if not removed
> altogether.
>

Thank you for this proposal. I think this is a good initiative and there 
are many benefits with an approach like this. Besides the benefit with 
"informed constraints", we have the option to apply constraints with 
more precision (e.g. on track level). A constraint structure doesn't 
have to take all media types and all the users devices into 
consideration in one single call. This should lead to simpler constraint 
structures.

If we wanted to, this could also lift the limitation that one call to 
getUserMedia() is only capable of returning one track of each media type 
(due to the limitation in the constraint structure to specify 
constraints for several tracks of the same type).

With Best-effort getUserMedia() we can also prevent a page from silently 
probing device capabilities without the user knowing it.

> Thanks!
>
> -------------
>
> 1. Create a suitable home for constraint application and capability
>     retrieval that is strongly tied to the track concept.
>
> In order to define a capabilities/constraints API that supports modifications,
> we need to carefully consider where it should live. I believe that it needs
> to exist only for "LocalMediaStreams" so that there is no confusion/ambiguity
> surrounding cloned MediaStreams or MediaStream objects obtained from a
> PeerConnection (remotely over a network).
> Note that providing a signaling channel or any other notification framework
> for constraint changes or capability requests over the network is out of scope
> for this proposal.
>
> The existing tracks concept seems like a good place for this API. However,
> because tracks can be added/removed from various MediaStreams (and
> LocalMediaStreams), there's no real guarantee that the tracks you're dealing
> with in the LocalMediaStream are the ones that are related to your "local"
> device. It's also confusing for developers if some tracks have special "local"
> capabilities while others don't (in the same track list). This is a problem
> that needs a solution if a consistent and reliable method for interacting with
> just your "local" (approved) tracks is to be designed. I'll call this the "transient-
> track" problem.
>
> Note: I'm not totally satisfied with this proposal, because even LocalMediaStream
> objects are potentially transient, and if the developer loses the reference to the
> LocalMediaStream, then they're out of luck. It's also a little weird to have two
> instances of LocalMediaStreams (from two calls to getUserMedia), but have basically
> the same information provided in parallel between them. Perhaps that's not a bit deal.
> Feedback, as always, is welcome.
>
> A new LocalMediaStreamTrackList is defined in order to address the transient-track
> problem noted above. These track lists are different from MediaStreamTrackLists in
> one notable way:
>
> * There is no ability to remove/add tracks to these lists. The addition and removal of
> tracks in these lists are managed exclusively by the user agent.
>
> The new track list's purpose is to aggregate all the "local" media stream tracks
> together for enumeration, settings, and constraint application. It also serves
> as a convenient list of the "active" local devices that have been approved by the
> user.
>

If I understand correctly, then all LocalMediaStream instances have the 
same audioDevices and videoDevices lists. That means a particular 
LocalMediaStream may render some local audio and video (the tracks), but 
it may change settings on all granted devices. I think that's a bit 
weird and it's not clear how the tracks in audioDevices relate to the 
tracks in audioTracks. The elements in audioDevices are more like global 
device controllers than playable tracks.

An example that shows the lack of relation between the different track 
lists is when I have localStream0, localStream1 and have just used 
getUserMedia() to create localStream2 which has access to new devices. 
As a result, an "addtrack" event is fired on the device lists in 
localStream0 and localStream1 for no obvious reason.

It would perhaps make more sense to move the device lists to a singleton 
object in the page (as discussed in [3]) and skip the track inheritance. 
Like e.g.:

navigator.userMedia.audioDevices; (move gUM to the userMedia object?
or
navigator.userAudioDevices;

In that case, the objects in the device lists should be of some new 
UserMediaDevice type and not inherit from MediaStreamTrack IMO.

However, I agree with what you say above that these settings should be 
on track level. The approach with a devices lists on a singleton object 
doesn't help to make it clearer which tracks are affected when you 
change settings on a device. One approach to make the connection between 
tracks and the settings objects clearer would be to make 
LocalMediaStreamTrack objects behave like real tracks. I think it's best 
described by idls:

interface AbstractMediaStream {
     readonly attribute DOMString label;
              attribute boolean ended;
              attribute EventHandler onended;
};

[Constructor (TracksUnionType? trackContainers)]
interface MediaStream : AbstractMediaStream {
     readonly attribute MediaStreamTrackList audioTracks;
     readonly attribute MediaStreamTrackList videoTracks;
};

interface LocalMediaStream : AbstractMediaStream {
     readonly attribute LocalMediaStreamTrackList audioTracks;
     readonly attribute LocalMediaStreamTrackList videoTracks;

     void stop();
};

There are only one track list (for each media type) in a media stream 
and it represents the playable tracks. Depending on the type of a 
stream, the tracks expose the appropriate settings API directly.

This model can also be extended to provide settings on aggregated, 
cloned and remotely produced streams that are not available in local 
streams.

Thanks again.

/Adam

> // +++ New interface
> interface LocalMediaStreamTrackList {
>     readonly attribute unsigned long length;
>     getter MediaStreamTrack (unsigned long index);
>     attribute EventHandler onaddtrack;
>     attribute EventHandler onremovetrack;
> };
>
> The developer can be notified when new local audio/video devices are enabled
> by the user (by subsequent calls to getUserMedia for example) or when local audio/
> video devices have stopped by registering for the [familiar] onaddtrack/onremovetrack
> event handlers.
>
> There are not add/remove APIs, as the management of this list is done exclusively
> by the user agent as previously noted.
>
> The LocalMediaStreamTrackList is surfaced on the LocalMediaStream interface as two
> new properties:
>
> // Existing definition...
> interface LocalMediaStream: MediaStream {
>     // Existing stop API
>     void stop ();
>     // +++ New
>     readonly attribute LocalMediaStreamTrackList audioDevices;
>     readonly attribute LocalMediaStreamTrackList videoDevices;
> };
>
> Each audio/video LocalMediaStreamTrackList is kept up-to-date among all
> LocalMediaStream instances, so that it doesn't matter which instance of a
> LocalMediaStream object is used, it will always have the aggregate information
> for all active local audio/video devices [previously] approved by the user.
>
> These two lists now represent (conceptually) the set of active devices
> supplying media streams on the user's local box. Each track within these
> lists essentially represents a local device. So these lists are an enumeration
> of the users active approved devices (not necessarily all the devices that
> the user has available). Since these are approved devices, it follows that
> requesting capabilities of and applying constraints to these devices is
> acceptable without requesting additional permissions from the user.
>
> Note that my proposal as-is doesn't describe how the developer can discover
> that there are more devices available.
>
> 2. Get the capabilities in terms of Track objects.
>
> In the proposal above, notice that the LocalMediaStreamTrackList contains a
> list of MediaStreamTrack objects. As already proposed by Rich in [2], I also
> recommend a factoring of MediaStreamTrack into two local derived types:
> LocalVideoStreamTrack and LocalAudioStreamTrack (my names include the prefix
> "local" for clarity):
>
> // +++ new factored interface for video-specific APIs
> interface LocalVideoStreamTrack : MediaStreamTrack {
> }
>
> // +++ new factored interface for audio-specific APIs
> interface LocalAudioStreamTrack : MediaStreamTrack {
> }
>
> The most-derived interfaces are always returned from the LocalMediaStreamTrackList.
>
> The current state of the track can be reflected in APIs added to these objects.
> As described in [2], for the LocalVideoStreamTrack these might include:
> * autoFocusMode
> * currentZoom
> * currentFlashMode
> * currentDisplayOrientation
> * viewingAngle
> * etc.
> LocalAudioStreamTracks might have:
> * currentAudioLevel
>
> Each local track contains the "current" settings, which can be directly
> inspected. In order to find the "range" of the available settings,
> e.g., the capabilities of a given track, I propose a common API across
> both track types:
>
> typedef sequence<MediaTrackConstraint>? LocalDeviceCapabilites;
> LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
>
> This API is located on the *track list* (i.e., LocalMediaStreamTrackList)
> in order to be able to operate on either a single track (e.g., "give me the
> capabilities for a single device") or all the available devices (e.g., "give
> me the capabilities for all the available devices"). (By "available", I mean
> those devices already approved by getUserMedia).
>
> // New interface
> interface LocalMediaStreamTrackList {
>     readonly attribute unsigned long length;
>     getter MediaStreamTrack (unsigned long index);
>     attribute EventHandler onaddtrack;
>     attribute EventHandler onremovetrack;
>     // +++ capabilities API
>     LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
> };
>
> If no parameter is provided, then the combined capabilities of all available tracks
> for the given list (audio/video) is returned. If the developer only has a single track,
> the API is very simple:
>
> var caps = localStream.videoDevices.getCapabilities();
>
> If there are multiple video devices currently active, then the developer
> uses the same code as above to get all the combined capabilities, or they can
> pick-and-choose:
>
> // Get the capabilities for the second active video device
> var caps = localStream.videoDevices.getCapabilities(1);
>
> The capabilities returned are the set of capability "ranges" that the device supports,
> suitable for use in the constraints API.
>
> 3. Apply constraints to Track objects
>
> Now that the capabilities can be inspected for approved devices (all or per-track),
> constraints can be applied to them. As with getting the capabilities, constraint
> application is applied either directly to a MediaStreamTrack or to all currently
> active local tracks in a given list (audio/video).
>
> // New interface
> interface LocalMediaStreamTrackList {
>     readonly attribute unsigned long length;
>     getter MediaStreamTrack (unsigned long index);
>     attribute EventHandler onaddtrack;
>     attribute EventHandler onremovetrack;
>     LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or unsigned long) track);
>     // +++ Constraints API
>     void applySettings(MediaTrackConstraints, optional (MediaStreamTrack or unsigned long) track);
> };
>
> I'm not a believer in track object mutation. In other words, I don't believe that
> a track should be able to be "in-place" updated by applying a constraint to it.
> Rather, my view is that if there is a change that the device makes in response
> to the application of new constraints, old tracks are stopped and new track objects
> are created in response. This makes the application of constraints implicitly
> asynchronous, which prevents developers from taking a dependency on the "immediate
> application" of constraints to a given track which may not be possible for all devices.
>
> The "applySettings" API (I renamed it to sound more friendly), acts on all the local
> media tracks in the list by default, or targets only a specific track if one is
> indicated in the 2nd parameter.
>
> If targeted toward a specific track, the new constraints are evaluated against
> the current track settings, and if the current settings fall within the new
> constraints then no change is made. If the constraints affect the current settings
> of the track, then that track is stopped (and consequently removed from the
> LocalMediaStreamTrackList as a result) and if the device associated with that track
> can support the new constraints, then a new track is created and added to the
> LocalMediaStreamTrackList. (All these actions trigger the appropriate add/remove
> events.)
>
> If not targeted toward a specific track (applying to all the current active devices),
> the above algorithm is run for every track in the list.
>
> To summarize:
> This proposal adds two new track lists to local media streams which correspond to
> the "active" local devices supplying the tracks which have been approved by
> getUserMedia. The proposal incorporates a getCapabilities and applySettings API onto
> these tracks lists which can operate on the list as a whole or on individual tracks
> in the list. This proposal also recommends that individual local tracks be augmented
> with device-specific "current" settings, though the details of these settings were
> only alluded to via another proposal [2].
>
> [1] http://lists.w3.org/Archives/Public/public-media-capture/2012Jul/0069.html
> [2] http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0032.html
>

[3] 
http://lists.w3.org/Archives/Public/public-media-capture/2012Aug/0080.html

Received on Thursday, 23 August 2012 12:40:34 UTC