Re: maxHeight and maxWidth

On 2/17/2016 2:29 PM, Peter Thatcher wrote:
>
>
> On Wed, Feb 17, 2016 at 10:26 AM, Randell Jesup 
> <randell-ietf@jesup.org <mailto:randell-ietf@jesup.org>> wrote:
>
>     Ugh, sent from the wrong address...
>
>     On 2/16/2016 9:35 PM, Peter Thatcher wrote:
>>     While thinking about this so more, I thought of another
>>     difficulty: if resolution is degraded, should that degradation
>>     happen before or after the max?
>>
>>     In your use case with two layers (1 full, 1 max of 90 pixels
>>     tall), what happens when the RtpSender degrades the resolution? 
>>     Let's say it degrades a video frame an original height of 360
>>     pixels down to a height of 180 pixels. Clearly, the full layer is
>>     now 180 pixels high.  But what should it do with the
>>     max-height-90-pixel layer be?  Should it become 45 pixels tall or
>>     stay 90 pixels?
>
>     90.
>     ​
>
>
>
>>     If it should stay 90, then what happens if the video further
>>     degrades down to 1/4 of the original size, such that the full
>>     layer is also 90 pixels high?  Would they both be 90?
>
>     yes - but the engine running simulcast should (not spec language!)
>     be smart enough to shut off the "full" layer at that point (when
>     the two resolutions get "close enough".  Similarly, if you defined
>     a maxwidth/height layer of 640x480, and feed in a camera at full
>     res - the engine should be smart enough to: for HD input, provide
>     2 layers, one 640x480, one full HD, and if the camera will only
>     provide 640x480 to provide only a single layer.  If bandwidth
>     degrades enough, the engine can degrade that layer even further.
>
>     Similarly, if you provide a ScaleBy of 2, and the bandwidth
>     degrades enough, the top layer will (should be) be shut
>     off/starved (though you could react by cutting the top-layer
>     resolution, and the lower layer would get cur too).  If bandwidth
>     degrades further, you don't (shouldn't IMHO) turn on the top layer
>     at a lower res and move the lower layer to 1/2 that, you should
>     just degrade the lower layer.
>
>
>     This is part of the logic of the
>     bandwidth-allocation-and-resolution-adaption code that lives in
>     the browser.  It's not defined by the spec, but should do
>     something "reasonable" and not too unexpected given the inputs
>     it's processing.  We should *not* try to lock down or even
>     significantly constrain such algorithms; just provide them input
>     on what the application would like to get out of it.
>
>
>
>     The other way you could go here would be to allow applications to
>     make or overrule all these decisions. This would require defining
>     a complex API with a bunch of inputs (which would be hard to
>     standardize across browsers), and also to run the algorithm in a
>     worker since you can't have it waiting on GC - and even then you
>     might have some control-latency issues.  Lets not go there.... 
>     And lets not go to "lock down a precise algorithm about how the
>     browser/simulcast engine must react" (see jib and getUserMedia
>     constraints).
>
>
> ​ I believe you just said "the browser should do X" but "the browser 
> can do whatever it wants".  Is that correct?

The first answer ("90") was what it should do (not must do).  I.e. 90 is 
what a browser "should" select there.  I don't think the spec should 
mandate that.

I would implement (for something supplying scaledownby and max) that you 
apply scaledownby first, then apply max.  And as stated, if the layers 
get "too close", you shut down the higher layer (which is an independent 
issue).  Specifying only scaledownby you won't run into the "too close" 
case until you force the scaled-down resolution to something 
ugly/useless.  I.e. if you have scaledownby 8 and an HD input - great!  
1920x1080 and 240x138.  but if the input drops down to 320x240, then 
40x30 is likely not very useful - at some point, the engine should make 
the decision to only provide the base layer, and drop the higher one.  
This gets a bit confusing if you wait too long, as it gets really small 
and then jumps to a lot larger (when you shut off the top layer).

Perhaps the simulcast model you're thinking of is where the source 
doesn't change, and the layer sizes are basically fixed at start time 
(though that kinda obviates the need for scale factors), and when 
bandwidth drops too much, or load goes too high, you only turn off the 
top layer - nothing ever changes the "input resolution". However, the 
"input resolution" can and does change in a lot of scenarios: window 
capture (or even screen capture), if you apply load/bandwidth scales to 
the top layer, if you're forwarding another MediaStream from a 
peerconnection (those really can change!), canvas.captureStream(), 
video_element.captureStream(), someone changing the constraint on the 
track being fed in, etc.

-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email randell-ietf@jesup.org!  Way too much spam