RE: Revised Constraints modification API proposal

> From: Harald Alvestrand [mailto:harald@alvestrand.no]
> On 08/21/2012 10:56 PM, Travis Leithead wrote:
> > In order to define a capabilities/constraints API that supports
> > modifications, we need to carefully consider where it should live. I
> > believe that it needs to exist only for "LocalMediaStreams" so that
> > there is no confusion/ambiguity surrounding cloned MediaStreams or
> > MediaStream objects obtained from a PeerConnection (remotely over a
> network).
>
> I fell off the logic chain right here.
> 
> I believe there's a need to specify and modify constraints on media streams
> that are not obtained from GetUserMedia, so I believe the API needs to exist
> as a MediaStreamTrack API, not a LocalMediaStreamTrack API.
> However, the constraints that make sense may be different for the two
> cases.

I admit that I don't have a firm grasp on what constraints would need to be
applied/changed via a local client on a MediaStreamTrack obtained from a
remote client. This isn't really my area of expertise. Can't the local client just
signal (through some out-of-band signaling channel that I think is out-of-scope)
to the remote client that they should apply the settings/constraints on the remote
box? I'd love to talk in specifics if you have some examples.

If these settings/constraints are related to PeerConnection and streaming settings,
then perhaps the settings/constraint application API can also be applied to that object.


> > Note: I'm not totally satisfied with this proposal, because even
> > LocalMediaStream objects are potentially transient, and if the
> > developer loses the reference to the LocalMediaStream, then they're out
> > of luck.
>
> Why is this a problem? If a developer loses the reference to a
> LocalMediaStream, we have to assume that he's got no further interest in
> modifying it, don't we?

Perhaps. The design problem that I see is that my "audioDevices/videoDevices" lists 
really belong on some type of singleton instance (like the navigator object). Having
these lists on potentially multiple instances seems a little weird to me. But it's the best
I can come up with. They _could_ move to navigator, but then a developer might
expect them to be populated with tracks and not realize that there's a connection
between getUserMedia and the contents of these lists. :-(


> > // +++ New interface
> > interface LocalMediaStreamTrackList {
> >     readonly attribute unsigned long length;
> >     getter MediaStreamTrack (unsigned long index);
> >     attribute EventHandler onaddtrack;
> >     attribute EventHandler onremovetrack; };
>
> Is this object a process-wide singleton object? How do applications obtain a
> refererence to this object?

Basically, there are two "domain-wide" singleton instances of this track object,
one for the audioDevices property getter, and one for the videoDevices property
getter. I say domain-wide because that's generally the scope of the permissions
from the UA.

Applications get a reference to the above object (the LocalMediaStreamTrackList) by
requesting the "audioDevices" or "videoDevices" property from any LocalMediaStream.


> > I'm not a believer in track object mutation. In other words, I don't
> > believe that a track should be able to be "in-place" updated by applying a
> > constraint to it.
> > Rather, my view is that if there is a change that the device makes in
> > response to the application of new constraints, old tracks are stopped
> > and new track objects are created in response. This makes the
> > application of constraints implicitly asynchronous, which prevents
> > developers from taking a dependency on the "immediate application" of
> constraints to a given track which may not be possible for all devices.
>
> I think I'm not getting this.
> What happens to references to the old track in other MediaStreams when
> such a replacement happens? Do they silently become invalid, or does some
> other magic happen?

This design is open to change, as I mentioned in my response to Rich.

To help explain what I wrote above, consider a scenario. Let's say you've got a 
MediaStream instance called "allTracks" which has aggregated all the tracks you've 
received from both local and remote sources. You also have a "localStream" which
has only your local audio and video track from your single webcam.

The question is, what happens when you apply a settings (constraint) change to
"localStream's" video track via localStream.videoDevices.applySettings(..). Let's call
the targeted track "localTrack1".

I would expect all of the following to happen (perhaps in this order):
1. localTrack1's onended event would fire
2. localTrack1 would be removed from the localStream.videoDevices list
3. localStream.videoDevices' onremovetrack event would fire
4. The UA would apply the settings and create a new track: "localTrack2"
5. localTrack2 would be added to the localStream.videoDevices list
6. localStream.videoDevices' onaddtrack event would fire
<optionally>
7. The application would need to manually remove "localTrack1" from allTracks.videoTracks (since it stopped).
8. The application would need to manually add "localTrack2" into allTracks.videoTracks

As Rich noted, this all becomes must easier if the track itself just adjusts its 
setting internally, so I'm open to changing this behavior. Having non-mutable track
objects just seemed like a good idea to me.

Received on Wednesday, 22 August 2012 18:18:21 UTC