Re: Relation between a track's constraints/settings and how it is transmitted

On Jan 19, 2016, at 23:49, Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> wrote:
> 
> I think there should be some more clarifications made here. E.g.

[BA] Can you file an Issue?

> 
> - what is scaleResolutionDownBy scaling down from? Presumably the width 
> and height of the MediaStreamTrack that is the source, but that is not 
> said anywhere

[BA] yes, from the input track, and should be stated.

> - what happens if the source track suddenly changes width or height 
> (either because the application applies another set of constraints, or 
> because the UA determines that is a good thing that can be done within 
> the current mandatory constraints (perhaps none has been set))? Will 
> that immediately propagate to the encoded stream?

[BA] Generally yes, though the encoder resolution change might be delayed until an I-frame is sent.

> - what happens if the local video has a ridiculous aspect ratio (say 
> height 100 width 5000), can that always be supported by an encoder?

[BA] Possibly not, in which case an error would result.

> - likewise, if the local video has higher resolution than the encoder 
> can support, meaning a track being sent with scaleResolutionDownBy==1.0 
> will go out with lower resolution than the local track; will the 
> scaleResolutionDownBy==4.0 simulcast version scale from the local video 
> or from the encoded with scale factor 1.0?

[BA] Good question.  I would say the track, but then scaleResolutionDownBy cannot be obeyed, and there is currently no event to tell the application that happened.

> 
> - if the frameRate of the local track is lower than maxFramerate, what 
> takes precedence?

[BA] maxFramerate is a maximum so the frameRate from the local track is what is encoded.

> 
>> 
>> Just as an example: let's say I constrain a local track to height
>> 1080 and framerate 60. There is (as I read the spec) no text saying
>> the encoder should encode something that can be decoded to 1080@60;
>> the UA could at its discretion - even if it has a good network
>> connection and a lot of resources - encode to 360@15. And we have no
>> "overconstrained" event to tell the app (even though I guess it can
>> find out from the stats).
>> 
>> [BA] RTCRtpEncodingParameters.scaleResolutionDownBy has a default
>> value of 1.0 (see Section 5.2.3), so if scaleResolutionDownBy is not
>> set to an alternative value, the instruction to the encoder is to
>> attempt to encode the resolution supplied by the local track.
> 
> I think "instruction to the encoder is" is the key here, I don't find 
> text that says something like that in the document.
> 
>> 
>> If RTCRtpEncodingParameters.maxFramerate is not set, the encoder
>> should attempt to encode the framerate supplied by the local track
>> (since there is no maximum framerate indicating otherwise).
> 
> That is what I think should happen as well, but I don't think we have 
> any text saying that currently.
> 
>> 
>> In the event of congestion, it may not be possible for the encoder to
>> carry out the RTCRtpEncodingParameters instructions.  For example, if
>> scaleResolutionDownBy is set to 1.0, but
>> parameters.degradationPreference is set to "maintain-framerate", then
>> resolution will degrade first.  getParameters() will return
>> parameters.encodings[j].scaleResolutionDownBy set to 1.0.  Within the
>> Statistics API, RTCMediaStreamTrackStats.frameHeight and frameWidth
>> provide information on the resolution that is actually being sent.
>> Currently, is no "overconstrained" event for the encoder.
>> 
>> Similarly, if parameters.degradationPreference is set to
>> "maintain-resolution", in the event of congestion, framerate will
>> degrade first.  getParameters() will return the value of
>> Parameters.encodings[j].maxFramerate that was originally set.
>> Within the Statistics API, RTCMediaStreamTrackStats.framesPerSecond
>> provides the frames per second that are sent.
> 
> I agree to the above (and I think the spec is sufficiently clear here).
> 
>> 
>> Of course, congestion could get so bad that both framerate and
>> resolution degrades, regardless of the value of
>> parameters.degradationPreference.
>> 
>> Stefan also said: "So my question was basically if we should have
>> text saying the UA should (try to) honor the track constraints when
>> setting up encoders."
>> 
>> [BA] Feel free to suggest some text.
> 
> This is probably what I should do.
> 
>> 
>> 
>> 
>

Received on Wednesday, 20 January 2016 15:45:31 UTC