Re: Revised Constraints modification API proposal

On 08/22/2012 07:48 PM, Travis Leithead wrote:
>> From: Rich Tibbett [mailto:richt@opera.com]
>> Thanks for putting this together. Generally speaking there is a lot to like
>> here!
> [..]
>
>>> On Aug 21, 2012, at 10:56 PM, Travis Leithead
>>> <travis.leithead@microsoft.com> wrote:
>>> // Existing definition...
>>> interface LocalMediaStream: MediaStream {
>>>    // Existing stop API
>>>    void stop ();
>>>    // +++ New
>>>    readonly attribute LocalMediaStreamTrackList audioDevices;
>>>    readonly attribute LocalMediaStreamTrackList videoDevices; };
>> Separating out devices from tracks makes a lot of sense. I assume always that
>> audioTracks >= audioDevices (since each item in audioDevices also has a
>> representation in audioTracks) ?
> As I understand the current tracks concept, I don't think that audioTracks >=
> audioDevices can always hold true. It is certainly true initially, but consider this:
>
> 1. getUserMedia request (for audio)
> 2. LocalMediaStream ("localStream") obtained (audioTracks.length = 1, audioDevices.length = 1)
> 3. Fork a new MediaStream using the audio track:
>       new MediaStream(localStream.audioTracks);
> 4. Remove the forked track from the LocalMediaStream:
>       localStream.audioTracks.remove(localStream.audioTracks[0]);
> 5. Now the localStream has (audioTracks.length = 0, audioDevices = 1)
> 6. Ask for another audio device with getUserMedia
> 7. The returned 2nd instance of a localStream2 will have (audioTracks.length = 1, audioDevices.length = 2)
>
> Now, it's possible that I misunderstood how the UA should react to a second call
> to getUserMedia, and that the same LocalMediaStream object instance (singleton?)
> is supposed to be returned in this case, but that's not currently how I'm reading the
> spec.

getUserMedia() called twice is supposed to be a way of getting two 
different devices, according to previous discussions.

>
>
>>> Each audio/video LocalMediaStreamTrackList is kept up-to-date among
>>> all LocalMediaStream instances, so that it doesn't matter which
>>> instance of a LocalMediaStream object is used, it will always have the
>>> aggregate information for all active local audio/video devices [previously]
>> approved by the user.
>>
>> So the list of tracks returned from getUserMedia is mutable? That's a big
>> change to the current proposal IIUC.
> According to the current draft, the remove() and add() APIs can alter the
> list of tracks:
> http://dev.w3.org/2011/webrtc/editor/getusermedia.html#widl-MediaStreamTrackList-remove-void-MediaStreamTrack-track
> If I understand your question, that means that the track list is mutable. Is
> that not also your understanding of the spec?

I have previously argued that the remove() API is conceptually harmful 
(it changes the index of tracks in the list, rendering references in the 
form MS + index unstable). Currently, both exist.

>
>
>>> These two lists now represent (conceptually) the set of active devices
>>> supplying media streams on the user's local box. Each track within
>>> these lists essentially represents a local device. So these lists are
>>> an enumeration of the users active approved devices (not necessarily
>>> all the devices that the user has available). Since these are approved
>>> devices, it follows that requesting capabilities of and applying
>>> constraints to these devices is acceptable without requesting additional
>>> permissions from the user.
>>>
>>> Note that my proposal as-is doesn't describe how the developer can
>>> discover that there are more devices available.
>> We can probably leave this one to later. Initial thoughts are that we could add
>> event listeners that inform when the total number of cameras/microphones
>> available on the local device changes. A subsequent call to getUserMedia
>> would still be required though to obtain potential access to those devices. It's
>> in the developers best interests to say something like 'Hey, we notice you've
>> attached another camera! [Click here] if you want to use this camera'. At
>> which point getUserMedia can be called, the opt-in UI can be displayed and
>> the process repeats.
> Agree. We need to ensure that the process is well-defined, whatever it ends
> up becoming.
>
>
>>> typedef sequence<MediaTrackConstraint>? LocalDeviceCapabilites;
>>> LocalDeviceCapabilites getCapabilities(optional (MediaStreamTrack or
>>> unsigned long) track);
>>>
>>> This API is located on the *track list* (i.e.,
>>> LocalMediaStreamTrackList) in order to be able to operate on either a
>>> single track (e.g., "give me the capabilities for a single device") or
>>> all the available devices (e.g., "give me the capabilities for all the
>>> available devices"). (By "available", I mean those devices already approved
>> by getUserMedia).
>>
>> What if two different cameras/microphones have conflicting capabilities?
>> How would they be expressed in a single call to getCapabilities without any
>> arguments?
> Hmm. Great point. I suppose we could have slightly different return values from
> the API if there are multiple devices in the list (e.g., each device's capabilities
> would be enumerated separately). However, it may just be easier to narrow the
> scope of the API to always working against a single track.... and if we go that route,
> then perhaps this API can just move directly to the track objects themselves.
>
> My only reservation with moving the API to the track is that we'd need to have two
> instances of the API, one for each of the derived local track types:
> * LocalVideoStreamTrack
> * LocalAudioStreamTrack
> Then again, we could also further factor these down to inherit from a common:
> * LocalMediaStreamTrack
> ...upon which the settings/constraints could hang.
>
> I'm certainly open to other's thoughts on this. As Harald mentioned in his response,
> remote MediaStreamTracks may want to provide settings and have contraints applied.
> While this is somewhat outside my scope of expertise, it doesn't seem unreasable to
> just apply the settings/contraints APIs to the raw MediaStreamTrack interface. If we
> go that route, web developers will just have to know that the return values for the settings
> API (and parameters to the constraints API) will be different depending on whether the
> tracks are local or remote. This is somewhat mitigated by grouping all the local
> tracks into the audioDevices/videoDevices lists, I suppose.
>
>
>>> I'm not a believer in track object mutation. In other words, I don't
>>> believe that a track should be able to be "in-place" updated by applying a
>> constraint to it.
>>
>> I think I don't share this view :( Given a stream, that we pipe data through, it
>> seems reasonable for that pipe to change its characteristics on the fly.
>>
>> What I think is happening here is that we're using
>> onaddtrack/onremovetrack to signal that capabilities have been applied.
>> Perhaps that would be better with a callback argument in applySettings or via
>> an event listener on each individual LocalMediaStreamTrack.
> This sounds like a good idea. I don't have a feel for what the WG feels is a
> better design, but I'm open to pursuing either course. My design assumes
> that applications are not well prepared for dynamic changes in track contents/
> settings. My design also forces web developers to re-hookup tracks to downstream
> MediaStream objects in the event of a setting change, which is somewhat
> undesirable.
Using add/remove track to signal track property change means that we 
have no stable reference for "this camera, even though it changed 
properties". This seems like a bad design to me.

The addition of a new resource and the modification of properties on a 
resource seems to me to be different events.
>
>
>>> The "applySettings" API (I renamed it to sound more friendly), acts on
>>> all the local media tracks in the list by default, or targets only a
>>> specific track if one is indicated in the 2nd parameter.
>> Again, what would happen if settings are conflicting between two devices? It
>> seems better if settings can be queried and applied only per track. Applying
>> settings to multiple tracks via a single call feels like it could be an optimisation
>> rather than a strictly necessary addition.
> As mentioned previously, this is a good point, and can be simplified by
> limiting the API to only work against a single track.
If a track is mutable, the natural place to have the mutation API is on 
the track.
>
>
>> I believe this proposal is heading in the right direction. There are naturally still
>> a few dark corners we need to explore but, in general, I like it. I strongly
>> support the direction this API is moving in. I'd like to explore what settings
>> we intend to allow via the applySettings call and discuss further on media
>> stream track mutability.
> Yes, I'd like to get very specific about what settings are exposed and how they
> are set via applySettings.
>
> I might have addressed your questions regarding track mutability above. But
> let's continue to discuss if needed.
>
>
>> It feels like something Opera could actively support and I would be happy to
>> contribute time to put this in to draft spec form with you.
> This is wonderful news. Thanks for your support!
>
>

Received on Thursday, 23 August 2012 10:06:49 UTC