Re: Some questions about RTCRtpEncodingParameters...

Some implementations do not vary the base framerate or resolution. For those, the "min" variables do not constitute encoder guidance.

On May 11, 2014, at 23:09, "Justin Uberti" <juberti@google.com<mailto:juberti@google.com>> wrote:

Do we need them all right now? I would argue we could probably get by with a lot less for ORTC 1.


On Sat, May 10, 2014 at 1:03 PM, Robin Raymond <robin@hookflash.com<mailto:robin@hookflash.com>> wrote:

I'm aware of at  least 6 SVC modes of operation, but I believe the top 3 we should be concerned with are "spacial", "temporal" and "quality".

To support that you need to be able to have a minimal value per each supported mode:
- minScale - don't go smaller than this size
- minFrameRate - don't send lower than this frame rate
- minQuality - don't let quality go less than this

Those values would typically get used as the min requirements for a base layer in SVC.

Plus the "bias" doesn't work since it's only factoring "spacial" and "temporal" priorities but neglects "quality".

You need to have
- scale priority - relative to frame rate/quality
- frame rate priority - relative to scale/quality
- quality priority - relative to scale/frame rate

Then you need in each layer:
- relative scale - relative scale from base (how much of an enhancement is this layer)
- relative frame rate - relative frame rate from base (how much of an enhancement is this layer)
- relative quality - relative to base (how much of an enhancement is this layer)

You need to know what the purpose of each layer is: scale, frame rate, quality.

And there are cross dependencies between layers operating in different SVC modes.

(and yes, there are use cases why you need all of these if you you to support three SVC modes of operation)

-Robin

<compose-unknown-contact.jpg>
Peter Thatcher<mailto:pthatcher@google.com>
May 9, 2014 at 5:22 PM



On Fri, May 9, 2014 at 12:06 PM, Bernard Aboba <Bernard.Aboba@microsoft.com<mailto:Bernard.Aboba@microsoft.com>> wrote:
[BA] For example, while the encodingId and dependencyEncodingIds can be used to set up layering, to do layering and simulcast together requires setting up multiple sender objects.

[Peter] ​Why?  Can't you express a mix here?​

[BA] As an example, let's say that in a single sender object we wanted to simulcast two streams, each with different resolution, and each of which support temporal scalability with two layers.  So we have four streams overall.   Let's say we have a "framescale" variable which means
to use that fraction of the framerate from getUserMedia.  Would this work?

var encodings =[{
  layerId: "halfScaleBase",
  scale: 0.5,
  framescale: 0.5
}, {
  layerId: "fullScaleBase",
  scale: 1.0,
  framescale: 0.5
}, {
  layerId: "temporalEnhancemenToHalfScaleBase",
  layerDependencies: ["halfScaleBase"],
  scale: 0.5,
  framescale: 1.0
}, {
  layerId: "temporalEnhancementToFullScaleBase",
  layerDependencies: ["fullScaleBase"],
  scale: 1.0,
  framescale: 1.0
}]


​That seems like a reasonable way to express what you want.  Is it implementable?​  Also, I'm not sure if I like the name "framescale".  Perhaps "relativeFramerate" or "framerateScale"?



[Peter] That said, we do need to make sure that our RtpCapabilties object provides enough information to be able to do this.  Do you think it's lacking something in particular?​

[BA]  There are a few things that come to mind:

   a. One is a statement about what types of scalability a given encoder/decoder supports.  Another is how many layers they support within that type.    These can perhaps
   be combined into a capability array:  [temporalMaxLayers, spatialMaxLayers, qualityMaxLayers] where setting MaxLayers of a given type to 0 means "I don't support it at all".


​This:​

int temporalLayers;
int spatialLayers;
int qualityLayers;

sounds good to me, although I'm not sure about the quality layers, since we don't have a way to express those in RtpParameters yet (do we?).

Now where should these go?  Should they be in RtpCapabilities directly?  Or are they codec-specific, so we need to so something more fancy, like RtpVideoEncodingCapabilities { DOMString codecName;  int temporalLayers; ... }, and then have a sequence<RtpVideoEncodingCapabilities> in RtpCapabilities (much like there is an array of RtpEncodingParameters in RtpParameters)?



   b.  Another is a statement about what types of simulcast is supported within an object and how many streams are supported. This also might be expressed in a similar array.

int temporalLayers;
int spatialLayers;
​int simulcastLayers;

?

Although, I don't quite​ understand why there would be a limit.  You can create as many RtpSenders as you want, right?



[BA] Some other questions relating to whether these attributes make sense for SVC at all.
For example, priority makes sense for specifying the priority between audio and video – but what does it mean when it is specified in individual SVC layers?

​[Peter] Maybe it wouldn't.  But it does for simulcast, doesn't it?

[BA] Yes, for simulcast it makes sense.
​
[BA] Similarly, minQuality might make some sense for a base layer (as might minFrameRate or minResolution), but what does it mean to specify this at each SVC layer?

[Peter] ​Maybe for SVC it doesn't.  But it does for simulcast, doesn't it?

[BA] "Quality" has a specific meaning within SVC (e.g. "quality scalability").   So I'm not sure the variable is intended to have the same meaning in simulcast.

​I think for quality scalability​, we'd need a different field.  "qualityScale"?

​
[BA] Also, is it necessary to provide a maxBitrate knob for each layer in SVC?

​[Peter] Again, maybe not all knobs make sense for SVC.  So if you're doing SVC, don't use the knobs that don't make sense.​

[BA] The knobs may make sense at an overall level -- as in "I'd like to impose a maximum Bit rate on the combination of layers".

​
That last bit about a  maximum for all layers, I'm not sure how to solve.  I think Justin had an idea about how to do it previously.  Justin?  Are you reading this thread?​