Re: Some questions about RTCRtpEncodingParameters...

Do we need them all right now? I would argue we could probably get by with
a lot less for ORTC 1.


On Sat, May 10, 2014 at 1:03 PM, Robin Raymond <robin@hookflash.com> wrote:

>
> I'm aware of at  least 6 SVC modes of operation, but I believe the top 3
> we should be concerned with are "spacial", "temporal" and "quality".
>
> To support that you need to be able to have a minimal value per each
> supported mode:
> - minScale - don't go smaller than this size
> - minFrameRate - don't send lower than this frame rate
> - minQuality - don't let quality go less than this
>
> Those values would typically get used as the min requirements for a base
> layer in SVC.
>
> Plus the "bias" doesn't work since it's only factoring "spacial" and
> "temporal" priorities but neglects "quality".
>
> You need to have
> - scale priority - relative to frame rate/quality
> - frame rate priority - relative to scale/quality
> - quality priority - relative to scale/frame rate
>
> Then you need in each layer:
> - relative scale - relative scale from base (how much of an enhancement is
> this layer)
> - relative frame rate - relative frame rate from base (how much of an
> enhancement is this layer)
> - relative quality - relative to base (how much of an enhancement is this
> layer)
>
> You need to know what the purpose of each layer is: scale, frame rate,
> quality.
>
> And there are cross dependencies between layers operating in different SVC
> modes.
>
> (and yes, there are use cases why you need all of these if you you to
> support three SVC modes of operation)
>
> -Robin
>
>    Peter Thatcher <pthatcher@google.com>
>  May 9, 2014 at 5:22 PM
>
>
>
> On Fri, May 9, 2014 at 12:06 PM, Bernard Aboba <
> Bernard.Aboba@microsoft.com> wrote:
>
>> [BA] For example, while the encodingId and dependencyEncodingIds can be
>> used to set up layering, to do layering and simulcast together requires
>> setting up multiple sender objects.
>>
>> [Peter] ​Why?  Can't you express a mix here?​
>>
>> [BA] As an example, let's say that in a single sender object we wanted to
>> simulcast two streams, each with different resolution, and each of which
>> support temporal scalability with two layers.  So we have four streams
>> overall.   Let's say we have a "framescale" variable which means
>> to use that fraction of the framerate from getUserMedia.  Would this work?
>>
>> var encodings =[{
>>   layerId: "halfScaleBase",
>>   scale: 0.5,
>>   framescale: 0.5
>> }, {
>>   layerId: "fullScaleBase",
>>   scale: 1.0,
>>   framescale: 0.5
>> }, {
>>   layerId: "temporalEnhancemenToHalfScaleBase",
>>   layerDependencies: ["halfScaleBase"],
>>   scale: 0.5,
>>   framescale: 1.0
>> }, {
>>   layerId: "temporalEnhancementToFullScaleBase",
>>   layerDependencies: ["fullScaleBase"],
>>   scale: 1.0,
>>   framescale: 1.0
>> }]
>>
>>
> ​That seems like a reasonable way to express what you want.  Is it
> implementable?​  Also, I'm not sure if I like the name "framescale".
>  Perhaps "relativeFramerate" or "framerateScale"?
>
>
>
>>
>> [Peter] That said, we do need to make sure that our RtpCapabilties object
>> provides enough information to be able to do this.  Do you think it's
>> lacking something in particular?​
>>
>> [BA]  There are a few things that come to mind:
>>
>>    a. One is a statement about what types of scalability a given
>> encoder/decoder supports.  Another is how many layers they support within
>> that type.    These can perhaps
>>    be combined into a capability array:  [temporalMaxLayers,
>> spatialMaxLayers, qualityMaxLayers] where setting MaxLayers of a given type
>> to 0 means "I don't support it at all".
>>
>
>
> ​This:​
>
> int temporalLayers;
> int spatialLayers;
> int qualityLayers;
>
> sounds good to me, although I'm not sure about the quality layers, since
> we don't have a way to express those in RtpParameters yet (do we?).
>
> Now where should these go?  Should they be in RtpCapabilities directly?
>  Or are they codec-specific, so we need to so something more fancy, like
> RtpVideoEncodingCapabilities { DOMString codecName;  int temporalLayers;
> ... }, and then have a sequence<RtpVideoEncodingCapabilities> in
> RtpCapabilities (much like there is an array of RtpEncodingParameters in
> RtpParameters)?
>
>
>
>>
>>    b.  Another is a statement about what types of simulcast is supported
>> within an object and how many streams are supported. This also might be
>> expressed in a similar array.
>>
>
> int temporalLayers;
> int spatialLayers;
> ​int simulcastLayers;
>
> ?
>
> Although, I don't quite​ understand why there would be a limit.  You can
> create as many RtpSenders as you want, right?
>
>
>
>>
>> [BA] Some other questions relating to whether these attributes make sense
>> for SVC at all.
>> For example, priority makes sense for specifying the priority between
>> audio and video – but what does it mean when it is specified in individual
>> SVC layers?
>>
>> ​[Peter] Maybe it wouldn't.  But it does for simulcast, doesn't it?
>>
>> [BA] Yes, for simulcast it makes sense.
>> ​
>> [BA] Similarly, minQuality might make some sense for a base layer (as
>> might minFrameRate or minResolution), but what does it mean to specify this
>> at each SVC layer?
>>
>> [Peter] ​Maybe for SVC it doesn't.  But it does for simulcast, doesn't it?
>>
>> [BA] "Quality" has a specific meaning within SVC (e.g. "quality
>> scalability").   So I'm not sure the variable is intended to have the same
>> meaning in simulcast.
>>
>
> ​I think for quality scalability​, we'd need a different field.
>  "qualityScale"?
>
> ​
>> [BA] Also, is it necessary to provide a maxBitrate knob for each layer in
>> SVC?
>>
>> ​[Peter] Again, maybe not all knobs make sense for SVC.  So if you're
>> doing SVC, don't use the knobs that don't make sense.​
>>
>> [BA] The knobs may make sense at an overall level -- as in "I'd like to
>> impose a maximum Bit rate on the combination of layers".
>>
>> ​
> That last bit about a  maximum for all layers, I'm not sure how to solve.
>  I think Justin had an idea about how to do it previously.  Justin?  Are
> you reading this thread?​
>
>