Re: Connecting devices and tracks

On 11/09/2013 03:55 PM, Stefan Håkansson LK wrote:
> On 08/11/13 20:48, Martin Thomson wrote:
>> I was asked to write this up and I've sat on this for a while, worried
>> that the barrage of detail would destroy the message.  There's a fair
>> degree of nuance in the proposal, so I recommend reading through,
>> maybe skipping the code for getUserMedia() and then we can discuss the
>> various issues that you see.
>>
>> # Connecting Devices and Tracks
>>
>> The device enumeration function provided by getMediaDevices() is
>> critical to this proposal.  The object that is produced by
>> getMediaDevices(), MediaDeviceInfo, is extended to include a way to
>> open and use that source (we can entertain a name change later; it
>> doesn't appear in the API explicitly, but it might be nice for a
>> better handle):
>>
>>    partial interface MediaDeviceInfo {
>>      MediaStreamTrack createTrack(Constraints? constraints);
>>      void connectTrack(MediaStreamTrack track);
>>    }
>>
>> For someone who was going to be using the sourceId constraint to
>> select a source, this is really easy to use.
>>
>>    var track = deviceInfo.createTrack({ ... constraints ... });
>>
>> It gets a little harder when you don't want to choose, and for that
>> purpose I'm going to propose that we retain getUserMedia
>> (implementation below).
>>
>> The constructor of MediaStreamTrack becomes more critical to the
>> process.  The track that is acquired as a result of calling the
>> constructor directly produces black or silence.  You'll notice that
>> this is intentionally synchronous; there are consequences for that,
>> but I think that this is something that the browser will have to
>> handle.
>>
>> Any user consent interactions are triggered when connectTrack() is
>> called.  Once an active (i.e., unmuted) device is attached to the
>> track, the 'onunmuted' event is fired for that track.
>>
>> createTrack() is a shortcut for both creating a track and connecting it:
>>
>>    MediaDeviceInfo.createTrack = function createTrack(constraints) {
>>      var track = new {Video,Audio}MediaStreamTrack(constraints);
>>      this.connectTrack(track);
>>      return track;
>>    };
> Let's see if I've understood correctly:
>
> * If you do createTrack, the UA will immediately create a track 
> connected to the source in question, but it will only produce 
> silence/blackness until the user approves its use, right? So it could be 
> used immediately for negotiation?
>
> Is that about right?
>
> * What is then the use of the MediaStreamTrack constructor? It has been 
> argued that if you just construct a track (with no info at all about an 
> eventual source) it can't be used for negotiation, and I can see no 
> other use for it?
>
>
> * Is connectTrack supposed to give an error if the track you supply 
> already has a source, or is the plan to allow for changing source of a 
> track?
>
> I'm trying to understand what this proposal (which I like BTW, but if we 
> can get support the same functionality with fewer changes that feels 
> safer) buys us over the one Jan-Ivar did a long time ago [1].

Thanks Martin!

I see this as a fleshed-out version of what Jan-Ivar proposed (the link
you gave didn't have any actual spec language).

The challenge I set Martin was to create something that did not change
the syntax or semantics of getUserMedia, yet allowed getUserMedia to be
implemented as a Javascript function.

There are some missing code pieces (no "else" branch in the
trackConnected callback, for instance), but overall, I think it's close.

One property this has is that the sequence of entries in "devices"
controls the selection of device; in the original constraint-based
selection algorithm, the device that satisfied the longest list of
optional constraints would "win".

Perhaps we should think of adding a "rankOnConstraints" function to
deviceInfo?

>
>
>
> [1] 
> http://lists.w3.org/Archives/Public/public-media-capture/2013Sep/0080.html
>
>> And that's it.
>>
>> The following code demonstrates how this can be used to replicate the
>> behaviour of getUserMedia.  It uses createTrack().
>>
>> --
>>
>> navigator.getUserMedia  = function getUserMedia(constraints,
>> successCb, failureCb) {
>>    // error checking omitted
>>    if (constraints.video) {
>>      getTrack("video", constraints.video, trackConnected, trackFailure);
>>    }
>>    if (constraints.audio) {
>>      getTrack("audio", constraints.audio, trackConnected, trackFailure);
>>    }
>>
>>    var firstTrack;
>>    function trackConnected(track) {
>>      if (firstTrack || !constraints.video || !constraints.audio) {
>>         successCb(new MediaStream([firstTrack, track]));
>>      }
>>    }
>>
>>    var failedAlready;
>>    function trackFailure(e) {
>>      if (!failedAlready) {
>>        failedAlready = true;
>>        failureCb(e);
>>      }
>>    }
>> };
>>
>> function getTrack(type, constraints, successCb, failureCb) {
>>    var devices = navigator.getMediaDevices().filter(ofType(type));
>>
>>    function stopMonitoring(track) {
>>      track.removeEventListener('unmute', connected); // overly cautious here
>>      track.removeEventListener('overconstrained', tryDevice);
>>    }
>>
>>    function tryDevice(e) {
>>      if (e) {
>>        if (timeout) {
>>          window.clearTimeout(timeout);
>>        }
>>        stopMonitoring(e.target);
>>        e.target.stop();
>>      }
>>
>>      var dev = devices.shift(); // pick the next one
>>      if (!dev) {
>>        failureCb(new Error('overconstrained'));  // use real thing
>>      }
>>      var track = dev.openTrack(typeof constraints === 'object' ?
>> constraints : undefined);
>>      track.addEventListener('unmute', connected);
>>      track.addEventListener('overconstrained', tryDevice);
>>
>>      var timeout = window.setTimeout(tryDevice, 1000);
>>    }
>>
>>    function connected(e) {
>>      stopMonitoring(e.target);
>>      successCb(e.target);
>>    }
>> }
>>
>> function ofType(t) {
>>    return function(x) {
>>      return x.type == "video";
>>    };
>> }
>>
>> --
>>
>> Please excuse the inevitable bugs and syntax errors; obviously I can't
>> compile or run this and I didn't bother linting it.
>>
>> Obviously, this isn't going to be how a browser would really implement
>> this.  This results in the camera going on all cameras in turn, which
>> is not ideal.  It might also find a device that meets the provided
>> contraints but is suboptimal. Another problem is that this requires a
>> timeout in case a muted device is selected because we don't
>> distinguish between muted and no user consent (intentionally).  A
>> browser can, of course, use the extra knowledge it has about sources
>> in order to implement these functions.
>>
>> Names are, of course, negotiable, as is every other aspect of this.
>> For instance, I was considering the use of an overload on
>> "connectTrack", but the difference in method signatures seemed enough
>> to justify two names.
>>
>>
>


-- 
Surveillance is pervasive. Go Dark.

Received on Saturday, 9 November 2013 21:34:50 UTC