RE: Revised Constraints modification API proposal

> From: Rich Tibbett [mailto:richt@opera.com]
> Thanks for putting this together. Generally speaking there is a lot to like
> here!

[..]

> > On Aug 21, 2012, at 10:56 PM, Travis Leithead
>> <travis.leithead@microsoft.com> wrote:
> > // Existing definition...
> > interface LocalMediaStream: MediaStream {
> >   // Existing stop API
> >   void stop ();
> >   // +++ New
> >   readonly attribute LocalMediaStreamTrackList audioDevices;
> >   readonly attribute LocalMediaStreamTrackList videoDevices; };
> 
> Separating out devices from tracks makes a lot of sense. I assume always that
> audioTracks >= audioDevices (since each item in audioDevices also has a
> representation in audioTracks) ?

As I understand the current tracks concept, I don't think that audioTracks >= 
audioDevices can always hold true. It is certainly true initially, but consider this:

1. getUserMedia request (for audio)
2. LocalMediaStream ("localStream") obtained (audioTracks.length = 1, audioDevices.length = 1)
3. Fork a new MediaStream using the audio track:
     new MediaStream(localStream.audioTracks);
4. Remove the forked track from the LocalMediaStream:
     localStream.audioTracks.remove(localStream.audioTracks[0]);
5. Now the localStream has (audioTracks.length = 0, audioDevices = 1)
6. Ask for another audio device with getUserMedia
7. The returned 2nd instance of a localStream2 will have (audioTracks.length = 1, audioDevices.length = 2)

Now, it's possible that I misunderstood how the UA should react to a second call 
to getUserMedia, and that the same LocalMediaStream object instance (singleton?)
is supposed to be returned in this case, but that's not currently how I'm reading the
spec.


> > Each audio/video LocalMediaStreamTrackList is kept up-to-date among
> > all LocalMediaStream instances, so that it doesn't matter which
> > instance of a LocalMediaStream object is used, it will always have the
> > aggregate information for all active local audio/video devices [previously]
> approved by the user.
> 
> So the list of tracks returned from getUserMedia is mutable? That's a big
> change to the current proposal IIUC.

According to the current draft, the remove() and add() APIs can alter the
list of tracks:
http://dev.w3.org/2011/webrtc/editor/getusermedia.html#widl-MediaStreamTrackList-remove-void-MediaStreamTrack-track
If I understand your question, that means that the track list is mutable. Is 
that not also your understanding of the spec?


> > These two lists now represent (conceptually) the set of active devices
> > supplying media streams on the user's local box. Each track within
> > these lists essentially represents a local device. So these lists are
> > an enumeration of the users active approved devices (not necessarily
> > all the devices that the user has available). Since these are approved
> > devices, it follows that requesting capabilities of and applying
> > constraints to these devices is acceptable without requesting additional
> > permissions from the user.
> >
> > Note that my proposal as-is doesn't describe how the developer can
> > discover that there are more devices available.
> 
> We can probably leave this one to later. Initial thoughts are that we could add
> event listeners that inform when the total number of cameras/microphones
> available on the local device changes. A subsequent call to getUserMedia
> would still be required though to obtain potential access to those devices. It's
> in the developers best interests to say something like 'Hey, we notice you've
> attached another camera! [Click here] if you want to use this camera'. At
> which point getUserMedia can be called, the opt-in UI can be displayed and
> the process repeats.

Agree. We need to ensure that the process is well-defined, whatever it ends
up becoming.


> > typedef sequence<MediaTrackConstraint>? LocalDeviceCapabilites;
> > LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or
> > unsigned long) track);
> >
> > This API is located on the *track list* (i.e.,
> > LocalMediaStreamTrackList) in order to be able to operate on either a
> > single track (e.g., "give me the capabilities for a single device") or
> > all the available devices (e.g., "give me the capabilities for all the
> > available devices"). (By "available", I mean those devices already approved
> by getUserMedia).
> 
> What if two different cameras/microphones have conflicting capabilities?
> How would they be expressed in a single call to getCapabilities without any
> arguments?

Hmm. Great point. I suppose we could have slightly different return values from
the API if there are multiple devices in the list (e.g., each device's capabilities 
would be enumerated separately). However, it may just be easier to narrow the
scope of the API to always working against a single track.... and if we go that route,
then perhaps this API can just move directly to the track objects themselves.

My only reservation with moving the API to the track is that we'd need to have two 
instances of the API, one for each of the derived local track types: 
* LocalVideoStreamTrack
* LocalAudioStreamTrack
Then again, we could also further factor these down to inherit from a common:
* LocalMediaStreamTrack
...upon which the settings/constraints could hang.

I'm certainly open to other's thoughts on this. As Harald mentioned in his response,
remote MediaStreamTracks may want to provide settings and have contraints applied.
While this is somewhat outside my scope of expertise, it doesn't seem unreasable to
just apply the settings/contraints APIs to the raw MediaStreamTrack interface. If we
go that route, web developers will just have to know that the return values for the settings
API (and parameters to the constraints API) will be different depending on whether the 
tracks are local or remote. This is somewhat mitigated by grouping all the local
tracks into the audioDevices/videoDevices lists, I suppose.


> > I'm not a believer in track object mutation. In other words, I don't
> > believe that a track should be able to be "in-place" updated by applying a
> constraint to it.
> 
> I think I don't share this view :( Given a stream, that we pipe data through, it
> seems reasonable for that pipe to change its characteristics on the fly.
> 
> What I think is happening here is that we're using
> onaddtrack/onremovetrack to signal that capabilities have been applied.
> Perhaps that would be better with a callback argument in applySettings or via
> an event listener on each individual LocalMediaStreamTrack.

This sounds like a good idea. I don't have a feel for what the WG feels is a 
better design, but I'm open to pursuing either course. My design assumes
that applications are not well prepared for dynamic changes in track contents/
settings. My design also forces web developers to re-hookup tracks to downstream
MediaStream objects in the event of a setting change, which is somewhat 
undesirable.


> > The "applySettings" API (I renamed it to sound more friendly), acts on
> > all the local media tracks in the list by default, or targets only a
> > specific track if one is indicated in the 2nd parameter.
> 
> Again, what would happen if settings are conflicting between two devices? It
> seems better if settings can be queried and applied only per track. Applying
> settings to multiple tracks via a single call feels like it could be an optimisation
> rather than a strictly necessary addition.

As mentioned previously, this is a good point, and can be simplified by 
limiting the API to only work against a single track.


> I believe this proposal is heading in the right direction. There are naturally still
> a few dark corners we need to explore but, in general, I like it. I strongly
> support the direction this API is moving in. I'd like to explore what settings
> we intend to allow via the applySettings call and discuss further on media
> stream track mutability.

Yes, I'd like to get very specific about what settings are exposed and how they
are set via applySettings.

I might have addressed your questions regarding track mutability above. But
let's continue to discuss if needed.


> It feels like something Opera could actively support and I would be happy to
> contribute time to put this in to draft spec form with you.

This is wonderful news. Thanks for your support!

Received on Wednesday, 22 August 2012 17:49:00 UTC