RE: Fifth iteration of Settings API now available

Seems this mail never made it to the list; resending (and sorry for spamming if both eventually are delivered). Stefan

-----Original Message-----
From: Stefan Hakansson LK [mailto:stefan.lk.hakansson@ericsson.com] 
Sent: den 5 december 2012 13:20
To: Travis Leithead
Cc: public-media-capture@w3.org
Subject: Re: Fifth iteration of Settings API now available

Comments in-line,

and I also have a question:

Is it possible to swap the source of a track? I've heard people asking for this functionality. I don't know if it is critical; if you want to swap the source you can already now create different (e.g. video) track, and then select the one you want to play the the media element.

Br,
Stefan

On 12/04/2012 07:55 PM, Travis Leithead wrote:
> Thanks for the feedback!
>
>> -----Original Message----- From: Stefan Hakansson LK 
>> [mailto:stefan.lk.hakansson@ericsson.com] Sent: Tuesday, December 4, 
>> 2012 4:34 AM To: public-media-capture@w3.org Subject: Re: Fifth 
>> iteration of Settings API now available
>>
>> Travis,
>>
>> many thanks for doing this work. There is a lot I like about it, but 
>> some comments below. Note: the things I don't comment I like - to me 
>> this seems well on the way to become baked enough for inclusion.
>>
>> 1. Detect new sources --------------------- In v4, new sources were 
>> reported to the application. I think this was very valuable - imagine 
>> me responding to an invitation immediately, using my lap-tops built 
>> in (crappy) camera. So while doing the first small-talk I manage to 
>> find and connect a better camera. Surely it would be good if the app 
>> became aware of the new camera and could use it? Similarly for audio, 
>> it usually takes me a minute or two to get my USB headset connected, 
>> but I would like to start using it once done. Is there any way to add 
>> this feature (and I am not insisting on it being there from day one, 
>> just that things are specified so it can be added)?
>
> There wasn't a great object to add this event to with the demise of 
> the device list. I considered using a callback style approach on the 
> source object constructors: static void observeDeviceAdded(callback 
> void ()) with its corresponding "disconnect" or stop observering 
> method.
>
> Alternatively, I could just add a new event handler to the navigator
> object: navigator.onusermediaadded
>
> If this feature is really needed then does the WG have a preference?
>
> Note, that this use case can be achieved by polling the numDevices 
> method at Regular intervals if necessary (once every minute or 30 
> seconds).

Perhaps that is sufficient. A removed (unplugged) device is already handled by the corresponding tracks becoming ended, and perhaps the app can regularly poll for new devices to detect if something is plugged in.

>
>
>
>> 2. source and remoteSource settings
>> ----------------------------------- Then I have a philosophical
>> thing I want to discuss a bit. As a background: I really like
>> section 4.1 and the discussion there. It clarifies a lot of things.
>> But, when you start digging in to the settings, it seems that (for
>> video) we end up in:
>>
>> * Width, Height * Framrate * Bitrate * rotation * mirror * zoom *
>> focusMode * fillLightMode
>>
>> You could reason like this in a situation where the video is
>> displayed at the other end of a PeerConnection: * Width and height
>> should be determined by the sink. If the video element is
>> 1920*1200, then this should propagate to the source. (Likewise if
>> the sink is a recorder). If there are several consumers, the
>> dimension of the largest one determines. * rotation and mirror can
>> be handled at display time (using css transforms). * zoom,
>> focusMode, fillLightMode and FrameRate are best handled at the
>> source - the source side would be in a better position to know if
>> light is needed, if the scene requires low or high framerate etc.
>> Note also that if these things can be set as remoteSource settings
>> there can be a conflict: what if one remoteSource asks for light
>> and another one (from another PeerConnection) asks for no light? (I
>> don't know if the same reasoning can be applied for
>> mirror/rotation)
>>
>> What I am getting at is a simplification: * remove rotation and
>> mirror from the settings - not needed. * remove width and height
>> from settings (at least from remoteSoource settings) and let the
>> consumer(s) control this * remove zoom, focusMode, fillLightMode
>> and FrameRate from remoteSource settings, maintain for localSource
>>
>> Not that the remote end could still control zoom, focusMode etc.
>> via in-app signaling if the app is designed that way - what we
>> remove is the need for signaling this using SDP o/a or in RTCP.
>>
>> This leaves bitrate; but to me bitrate is something that is
>> closely related to sending a MediaStreamTrack over a
>> PeerConnection. I would like this to be controlled at the sending
>> end of a PeerConnection along with things like priority rather than
>> at the receiving end. A reason is that you would like the initial
>> SDP offer to contain e.g. BW attributes.
>
> Interesting.
>
> You make a good argument for removing width/height as settings. I
> wonder you think it would be a good idea to leave the readonly
> attributes around so that the current width/height of the source
> could be extracted? But then again, if you can't change it (at least
> not via the source), then why have that data exposed?

The media element already exposes this data; perhaps it is good to know 
in certain situations along with e.g. framerate?


> The argument
> here that we need agreement on is: "do the sinks (and
> implementations) always know best?" If so, we probably should drop
> the corresponding constraint too, as users will need to apply this
> constraint to sinks, rather than sources, and there's already ways to
> do that (via HTML/CSS).

That's the question. But one thing is to ask for devices capable of a 
certain resolution, another is when the camera actually starts producing 
that resolution. To it seems reasonable that the camera does not produce 
any data until there is a sink of some sort - if for no other reason to 
save battery.

>
> One change we'd need to make in regard to the above comment, would be
> to ensure that a MediaRecorder (in Jim's proposal) can specify what
> width/height should be the target recording (as this will be
> necessary to ensure the source object supplies the right dimensions
> via the media stream).

Yes, precisely.

>
> I tend to agree with your thoughts on mirror and rotation.
>
> Zoom, focusMode, and fillLightMode are not on the remoteSource object
> to begin with. FrameRate is. I could remove frameRate.

"VideoRemoteStreamSource implements StreamSourceSettings", and the 
settings you can do include all the ones I mentioned. So the way I 
figured this was that there at the receiving end of a PeerConnection 
would be a StreamSourceSettings object associated with a ViderTrack 
being received.

Then it becomes really complex if the remote end can ask for "fillLight" 
and so one - that is the complexity I wanted to avoid.

> For the same
> reasons you argue above, would you not also want frameRate to be
> negotiated by the sinks, and otherwise provide the best frameRate
> possible?

Perhaps if the sink for some reason can not render the framerate the 
sender wants to use (e.g. not sufficient decoding resources). But the 
sender has better knowledge about what the camera will capture, and if 
that means high framerate is preferred. (I guess you could argue the 
other way as well). Basically I am open, but from a practical viewpoint 
there is something called "imageattr" that can be used to signal 
reslution, but I don't know if there is a "framerateattr". The fewer 
things that must be defined in signaling std (regardless of it is SDP or 
RTCP) the better probably....

>
> Regarding bitRate, are you suggesting that this doesn't belong as a
> setting on the source object? If it can already be provided directly
> to a Peer Connection, then it probably isn't needed.

Exactly, to me it belongs to the PeerConnection. And more specifically I 
think it belongs at the sending side (so that the PeerConn can produce 
SDP offers that contain the bitrate info).

>
>
>> 3. Detailed comments -------------------- a) For the
>> MediaStreamTrack definition, would it not be more natural that the
>> "id" was readonly (but that the app could assign a label of its
>> own)? And that the "id" must be maintained over a PeerConnection
>> (but maybe we don't need to signal label).
>
> Sounds good to me. Anyone object?
>
>
>> b) xxStreamSource/xxStreamRemoteSource/StreamSourceSettings: Should
>> we not substitute "Stream" for "Track" in all these cases?
>
> I have no problem with this.
>
>
>> c) VideoStreamTrack has a "facing" attribute. Does this not belong
>> to the VideoStreamSource?
>
> I've heard this from all of my reviewers so far. It must be true :-)
>
>
>> d) The VideoStreamSource has attributes like width, height, ... and
>> a method "getNumDevices". This does not really make sense to me,
>> the different devices could have different widths, heights and so
>> on set. To me it seems that getNumDevices (which should work for
>> audio too, right?), belongs elsewhere, but I may be getting
>> something wrong
>
> The key is the "static" flag. It means that:
> VideoStreamSource.getNumDevices == function VideoStreamSource.stop ==
> undefined mediaTrack.getNumDevices == undefined mediaTrack.stop ==
> function
>
> In short, it puts the method on the Constructor object and not that
> instance object. This means that the method can be invoked
> independently (without requiring an instance of a track object).

OK. THink I got it.

>
>
>> e) We would need to specify if/how the change of one setting alters
>> the preference order. IIUC the constraints structure, if used only
>> with optional constraints, establishes an order of preference. If
>> the first constraint is framerate it says that maintaining
>> framerate within the bounds is more important than the following
>> constraints. What happens if you later on change one that has lower
>> priority? Does it pop to the top of the list?
>
> Yes this needs to be defined. To avoid having to re-apply all the
> prior constraints just to maintain relative priority, there should be
> some more elegant way to in-line adjust a setting that was previously
> specified using the same relative priority, or to specify that a
> setting change trumps prior priority. Any suggestions for such a
> mechanism or different approaches are welcome...

I think Martin had a proposal here!

>
>
>> f) How should it be treated if a device can at setting time fulfill
>> the constraint (e.g. framerate) but later, due to e.g. overheating,
>> needs to drop below the range? Should a settingserror fire?
>
> This makes sense to me. To clarify, we need to define this for both
> optional and mandatory constraints when used as settings. I suspect
> that the firing of the settingserror event would only apply in
> situations where the mandatory settings are violated. Is the
> principle: "the media should continue streaming no matter what
> constraints need to be violated," or is it "the media should continue
> streaming until a mandatory constraint is violated" (then the stream
> can end)?

Or should it be: in the Mandatory case, the track simply stops (because 
that is what mandatory constraints are for, right?; in the optional a 
settingserror event fires.

>
>
>> g) We could consider removing "enabled" from the definition of the
>> MediaStreamTrack. Enabled/selected tracks can also be selected on
>> the media element, and having it on two places opens for
>> confusion.
>
> Also makes sense to me.
>
>
>> Stefan
>>
>> PS I think http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-
>> capture/proposals/SettingsAPI_proposal_v5.html changed a bit since
>> last time I looked. You could consider naming _v51. etc if you do
>> further changes so that referencing is not confused
>>
>>
>>
>>
>> On 12/01/2012 12:39 AM, Travis Leithead wrote:
>>> I've completed updates to a Settings API proposal v5.
>>>
>>> http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-
>> capture/proposals/SettingsAPI_proposal_v5.html
>>>
>>> The highlights: * Simplified settings model * Simplified
>>> mechanism for discovering other camera/mic devices (before
>> calling getUserMedia)
>>> * Dropped the device list concept * Simpler inheritance for
>>> tracks * Sources are a property of tracks * More complete
>>> definition of how settings are expected to work * Concepts
>>> introduced for constructable tracks and "new" readyState * New
>>> source objects introduced for tracks belonging to remote media
>> streams
>>>
>>> I've already received some feedback that I'll re-post here:
>>>
>>> * Some properties for settings may be better suited for the
>>> track
>> instead of the source (in v5 all settings are on source objects).
>> These include things like width/height on Video tracks.
>>> * Some properties of the tracks may be better suited for the
>>> source
>> object instead (like "facing").
>>> * The "get" API might be more useful if it returned the current
>> constraint (or constraint range) applied to the setting (since the
>> properties return the current value).
>>>
>>> -Travis
>>>
>>
>
>

Received on Wednesday, 5 December 2012 12:55:34 UTC