Re: Some questions about RTCRtpEncodingParameters...

On Fri, May 9, 2014 at 12:06 PM, Bernard Aboba
<Bernard.Aboba@microsoft.com>wrote:

> [BA] For example, while the encodingId and dependencyEncodingIds can be
> used to set up layering, to do layering and simulcast together requires
> setting up multiple sender objects.
>
> [Peter] ​Why?  Can't you express a mix here?​
>
> [BA] As an example, let's say that in a single sender object we wanted to
> simulcast two streams, each with different resolution, and each of which
> support temporal scalability with two layers.  So we have four streams
> overall.   Let's say we have a "framescale" variable which means
> to use that fraction of the framerate from getUserMedia.  Would this work?
>
> var encodings =[{
>   layerId: "halfScaleBase",
>   scale: 0.5,
>   framescale: 0.5
> }, {
>   layerId: "fullScaleBase",
>   scale: 1.0,
>   framescale: 0.5
> }, {
>   layerId: "temporalEnhancemenToHalfScaleBase",
>   layerDependencies: ["halfScaleBase"],
>   scale: 0.5,
>   framescale: 1.0
> }, {
>   layerId: "temporalEnhancementToFullScaleBase",
>   layerDependencies: ["fullScaleBase"],
>   scale: 1.0,
>   framescale: 1.0
> }]
>
>
​That seems like a reasonable way to express what you want.  Is it
implementable?​  Also, I'm not sure if I like the name "framescale".
 Perhaps "relativeFramerate" or "framerateScale"?



>
> [Peter] That said, we do need to make sure that our RtpCapabilties object
> provides enough information to be able to do this.  Do you think it's
> lacking something in particular?​
>
> [BA]  There are a few things that come to mind:
>
>    a. One is a statement about what types of scalability a given
> encoder/decoder supports.  Another is how many layers they support within
> that type.    These can perhaps
>    be combined into a capability array:  [temporalMaxLayers,
> spatialMaxLayers, qualityMaxLayers] where setting MaxLayers of a given type
> to 0 means "I don't support it at all".
>


​This:​

int temporalLayers;
int spatialLayers;
int qualityLayers;

sounds good to me, although I'm not sure about the quality layers, since we
don't have a way to express those in RtpParameters yet (do we?).

Now where should these go?  Should they be in RtpCapabilities directly?  Or
are they codec-specific, so we need to so something more fancy, like
RtpVideoEncodingCapabilities { DOMString codecName;  int temporalLayers;
... }, and then have a sequence<RtpVideoEncodingCapabilities> in
RtpCapabilities (much like there is an array of RtpEncodingParameters in
RtpParameters)?



>
>    b.  Another is a statement about what types of simulcast is supported
> within an object and how many streams are supported. This also might be
> expressed in a similar array.
>

int temporalLayers;
int spatialLayers;
​int simulcastLayers;

?

Although, I don't quite​ understand why there would be a limit.  You can
create as many RtpSenders as you want, right?



>
> [BA] Some other questions relating to whether these attributes make sense
> for SVC at all.
> For example, priority makes sense for specifying the priority between
> audio and video – but what does it mean when it is specified in individual
> SVC layers?
>
> ​[Peter] Maybe it wouldn't.  But it does for simulcast, doesn't it?
>
> [BA] Yes, for simulcast it makes sense.
> ​
> [BA] Similarly, minQuality might make some sense for a base layer (as
> might minFrameRate or minResolution), but what does it mean to specify this
> at each SVC layer?
>
> [Peter] ​Maybe for SVC it doesn't.  But it does for simulcast, doesn't it?
>
> [BA] "Quality" has a specific meaning within SVC (e.g. "quality
> scalability").   So I'm not sure the variable is intended to have the same
> meaning in simulcast.
>

​I think for quality scalability​, we'd need a different field.
 "qualityScale"?

​
> [BA] Also, is it necessary to provide a maxBitrate knob for each layer in
> SVC?
>
> ​[Peter] Again, maybe not all knobs make sense for SVC.  So if you're
> doing SVC, don't use the knobs that don't make sense.​
>
> [BA] The knobs may make sense at an overall level -- as in "I'd like to
> impose a maximum Bit rate on the combination of layers".
>
> ​
That last bit about a  maximum for all layers, I'm not sure how to solve.
 I think Justin had an idea about how to do it previously.  Justin?  Are
you reading this thread?​