RE: Fifth iteration of Settings API now available

Thanks for the feedback!

> -----Original Message-----
> From: Stefan Hakansson LK [mailto:stefan.lk.hakansson@ericsson.com]
> Sent: Tuesday, December 4, 2012 4:34 AM
> To: public-media-capture@w3.org
> Subject: Re: Fifth iteration of Settings API now available
> 
> Travis,
> 
> many thanks for doing this work. There is a lot I like about it, but
> some comments below. Note: the things I don't comment I like - to me
> this seems well on the way to become baked enough for inclusion.
> 
> 1. Detect new sources
> ---------------------
> In v4, new sources were reported to the application. I think this was
> very valuable - imagine me responding to an invitation immediately,
> using my lap-tops built in (crappy) camera. So while doing the first
> small-talk I manage to find and connect a better camera. Surely it would
> be good if the app became aware of the new camera and could use it?
> Similarly for audio, it usually takes me a minute or two to get my USB
> headset connected, but I would like to start using it once done. Is
> there any way to add this feature (and I am not insisting on it being
> there from day one, just that things are specified so it can be added)?

There wasn't a great object to add this event to with the demise of the device list. I considered using a callback style approach on the source object constructors:
static void observeDeviceAdded(callback void ())
with its corresponding "disconnect" or stop observering method.

Alternatively, I could just add a new event handler to the navigator object:
navigator.onusermediaadded

If this feature is really needed then does the WG have a preference?

Note, that this use case can be achieved by polling the numDevices method at
Regular intervals if necessary (once every minute or 30 seconds).


 
> 2. source and remoteSource settings
> -----------------------------------
> Then I have a philosophical thing I want to discuss a bit. As a
> background: I really like section 4.1 and the discussion there. It
> clarifies a lot of things. But, when you start digging in to the
> settings, it seems that (for video) we end up in:
> 
> * Width, Height
> * Framrate
> * Bitrate
> * rotation
> * mirror
> * zoom
> * focusMode
> * fillLightMode
> 
> You could reason like this in a situation where the video is displayed
> at the other end of a PeerConnection:
> * Width and height should be determined by the sink. If the video
> element is 1920*1200, then this should propagate to the source.
> (Likewise if the sink is a recorder). If there are several consumers,
> the dimension of the largest one determines.
> * rotation and mirror can be handled at display time (using css
> transforms).
> * zoom, focusMode, fillLightMode and FrameRate are best handled at the
> source - the source side would be in a better position to know if light
> is needed, if the scene requires low or high framerate etc. Note also
> that if these things can be set as remoteSource settings there can be a
> conflict: what if one remoteSource asks for light and another one (from
> another PeerConnection) asks for no light? (I don't know if the same
> reasoning can be applied for mirror/rotation)
> 
> What I am getting at is a simplification:
> * remove rotation and mirror from the settings - not needed.
> * remove width and height from settings (at least from remoteSoource
> settings) and let the consumer(s) control this
> * remove zoom, focusMode, fillLightMode and FrameRate from remoteSource
> settings, maintain for localSource
> 
> Not that the remote end could still control zoom, focusMode etc. via
> in-app signaling if the app is designed that way - what we remove is the
> need for signaling this using SDP o/a or in RTCP.
> 
> This leaves bitrate; but to me bitrate is something that is closely
> related to sending a MediaStreamTrack over a PeerConnection. I would
> like this to be controlled at the sending end of a PeerConnection along
> with things like priority rather than at the receiving end. A reason is
> that you would like the initial SDP offer to contain e.g. BW attributes.

Interesting.

You make a good argument for removing width/height as settings. I wonder you think it would be a good idea to leave the readonly attributes around so that the current width/height of the source could be extracted? But then again, if you can't change it (at least not via the source), then why have that data exposed? The argument here that we need agreement on is: "do the sinks (and implementations) always know best?" If so, we probably should drop the corresponding constraint too, as users will need to apply this constraint to sinks, rather than sources, and there's already ways to do that (via HTML/CSS).

One change we'd need to make in regard to the above comment, would be to ensure that a MediaRecorder (in Jim's proposal) can specify what width/height should be the target recording (as this will be necessary to ensure the source object supplies the right dimensions via the media stream).

I tend to agree with your thoughts on mirror and rotation.

Zoom, focusMode, and fillLightMode are not on the remoteSource object to begin with. FrameRate is. I could remove frameRate. For the same reasons you argue above, would you not also want frameRate to be negotiated by the sinks, and otherwise provide the best frameRate possible?

Regarding bitRate, are you suggesting that this doesn't belong as a setting on the source object? If it can already be provided directly to a Peer Connection, then it probably isn't needed.


> 3. Detailed comments
> --------------------
> a) For the MediaStreamTrack definition, would it not be more natural
> that the "id" was readonly (but that the app could assign a label of its
> own)? And that the "id" must be maintained over a PeerConnection (but
> maybe we don't need to signal label).

Sounds good to me. Anyone object?


> b) xxStreamSource/xxStreamRemoteSource/StreamSourceSettings: Should we
> not substitute "Stream" for "Track" in all these cases?

I have no problem with this.


> c) VideoStreamTrack has a "facing" attribute. Does this not belong to
> the VideoStreamSource?

I've heard this from all of my reviewers so far. It must be true :-)


> d) The VideoStreamSource has attributes like width, height, ... and a
> method "getNumDevices". This does not really make sense to me, the
> different devices could have different widths, heights and so on set. To
> me it seems that getNumDevices (which should work for audio too,
> right?), belongs elsewhere, but I may be getting something wrong

The key is the "static" flag. It means that:
VideoStreamSource.getNumDevices == function
VideoStreamSource.stop == undefined
mediaTrack.getNumDevices == undefined
mediaTrack.stop == function

In short, it puts the method on the Constructor object and not that instance object. This means that the method can be invoked independently (without requiring an instance of a track object).


> e) We would need to specify if/how the change of one setting alters the
> preference order. IIUC the constraints structure, if used only with
> optional constraints, establishes an order of preference. If the first
> constraint is framerate it says that maintaining framerate within the
> bounds is more important than the following constraints. What happens if
> you later on change one that has lower priority? Does it pop to the top
> of the list?

Yes this needs to be defined. To avoid having to re-apply all the prior constraints just to maintain relative priority, there should be some more elegant way to in-line adjust a setting that was previously specified using the same relative priority, or to specify that a setting change trumps prior priority. Any suggestions for such a mechanism or different approaches are welcome...


> f) How should it be treated if a device can at setting time fulfill the
> constraint (e.g. framerate) but later, due to e.g. overheating, needs to
> drop below the range? Should a settingserror fire?

This makes sense to me. To clarify, we need to define this for both optional and mandatory constraints when used as settings. I suspect that the firing of the settingserror event would only apply in situations where the mandatory settings are violated. Is the principle: "the media should continue streaming no matter what constraints need to be violated," or is it "the media should continue streaming until a mandatory constraint is violated" (then the stream can end)?


> g) We could consider removing "enabled" from the definition of the
> MediaStreamTrack. Enabled/selected tracks can also be selected on the
> media element, and having it on two places opens for confusion.

Also makes sense to me.


> Stefan
> 
> PS I think
> http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-
> capture/proposals/SettingsAPI_proposal_v5.html
> changed a bit since last time I looked. You could consider naming _v51.
> etc if you do further changes so that referencing is not confused
> 
> 
> 
> 
> On 12/01/2012 12:39 AM, Travis Leithead wrote:
> > I've completed updates to a Settings API proposal v5.
> >
> > http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-
> capture/proposals/SettingsAPI_proposal_v5.html
> >
> > The highlights:
> > * Simplified settings model
> > * Simplified mechanism for discovering other camera/mic devices (before
> calling getUserMedia)
> > * Dropped the device list concept
> > * Simpler inheritance for tracks
> > * Sources are a property of tracks
> > * More complete definition of how settings are expected to work
> > * Concepts introduced for constructable tracks and "new" readyState
> > * New source objects introduced for tracks belonging to remote media
> streams
> >
> > I've already received some feedback that I'll re-post here:
> >
> > * Some properties for settings may be better suited for the track
> instead of the source (in v5 all settings are on source objects). These
> include things like width/height on Video tracks.
> > * Some properties of the tracks may be better suited for the source
> object instead (like "facing").
> > * The "get" API might be more useful if it returned the current
> constraint (or constraint range) applied to the setting (since the
> properties return the current value).
> >
> > -Travis
> >
>

Received on Tuesday, 4 December 2012 18:56:09 UTC