Re: Relation between a track's constraints/settings and how it is transmitted

On 19/01/16 21:09, Bernard Aboba wrote:
> Stefan said:
>
> "I was actually at the step before. You are right in that constraints
> are applied on the track before it is input to the encoder. But I see
> no text saying that what the encoder does (and what is the
> characteristics of the track at the remote endpoint) does have to
> comply to that in any way.
>
> [BA] Section 5.2.4 defines the attributes within the
> RTCRtpEncodingParameters dictionary.  So there is some text
> describing what the encoder does - but perhaps not enough.

I think there should be some more clarifications made here. E.g.

- what is scaleResolutionDownBy scaling down from? Presumably the width 
and height of the MediaStreamTrack that is the source, but that is not 
said anywhere
- what happens if the source track suddenly changes width or height 
(either because the application applies another set of constraints, or 
because the UA determines that is a good thing that can be done within 
the current mandatory constraints (perhaps none has been set))? Will 
that immediately propagate to the encoded stream?
- what happens if the local video has a ridiculous aspect ratio (say 
height 100 width 5000), can that always be supported by an encoder?
- likewise, if the local video has higher resolution than the encoder 
can support, meaning a track being sent with scaleResolutionDownBy==1.0 
will go out with lower resolution than the local track; will the 
scaleResolutionDownBy==4.0 simulcast version scale from the local video 
or from the encoded with scale factor 1.0?

- if the frameRate of the local track is lower than maxFramerate, what 
takes precedence?

>
> Just as an example: let's say I constrain a local track to height
> 1080 and framerate 60. There is (as I read the spec) no text saying
> the encoder should encode something that can be decoded to 1080@60;
> the UA could at its discretion - even if it has a good network
> connection and a lot of resources - encode to 360@15. And we have no
> "overconstrained" event to tell the app (even though I guess it can
> find out from the stats).
>
> [BA] RTCRtpEncodingParameters.scaleResolutionDownBy has a default
> value of 1.0 (see Section 5.2.3), so if scaleResolutionDownBy is not
> set to an alternative value, the instruction to the encoder is to
> attempt to encode the resolution supplied by the local track.

I think "instruction to the encoder is" is the key here, I don't find 
text that says something like that in the document.

>
> If RTCRtpEncodingParameters.maxFramerate is not set, the encoder
> should attempt to encode the framerate supplied by the local track
> (since there is no maximum framerate indicating otherwise).

That is what I think should happen as well, but I don't think we have 
any text saying that currently.

>
> In the event of congestion, it may not be possible for the encoder to
> carry out the RTCRtpEncodingParameters instructions.  For example, if
> scaleResolutionDownBy is set to 1.0, but
> parameters.degradationPreference is set to "maintain-framerate", then
> resolution will degrade first.  getParameters() will return
> parameters.encodings[j].scaleResolutionDownBy set to 1.0.  Within the
> Statistics API, RTCMediaStreamTrackStats.frameHeight and frameWidth
> provide information on the resolution that is actually being sent.
> Currently, is no "overconstrained" event for the encoder.
>
> Similarly, if parameters.degradationPreference is set to
> "maintain-resolution", in the event of congestion, framerate will
> degrade first.  getParameters() will return the value of
> Parameters.encodings[j].maxFramerate that was originally set.
> Within the Statistics API, RTCMediaStreamTrackStats.framesPerSecond
> provides the frames per second that are sent.

I agree to the above (and I think the spec is sufficiently clear here).

>
> Of course, congestion could get so bad that both framerate and
> resolution degrades, regardless of the value of
> parameters.degradationPreference.
>
> Stefan also said: "So my question was basically if we should have
> text saying the UA should (try to) honor the track constraints when
> setting up encoders."
>
> [BA] Feel free to suggest some text.

This is probably what I should do.

>
>
>

Received on Wednesday, 20 January 2016 07:50:23 UTC