Re: Super-academic, highly-abstract meta-modelling time: The Media Path

Thanks for the positive feedback.

Collating replies...

On 14 February 2013 22:47, Harald Alvestrand <harald@alvestrand.no> wrote:
> A few additional observations:
>
> - I see multiple tracks connected to one source, with different incompatible constraints, as being a matter of attaching to "virtual configurations" of that single source. Thus, a camera being asked to feed at resoution min 1024x768 and max 150x100 at the same time isn't necessarily overconstrained; the two tracks are connected to different virtual configurations - which in practice will be achieved by inserting a downscaling step somewhere between the source and the min-requesting sink.

I considered this variation on the model, but that actually
complicates things somewhat.  If you consider min 1024x768 and max
150x100, the model accommodates that without any new virtual modes:
selection ignores the max 150x100, but that track requires a downscale
before the media is passed to its sink.

> - I think upscaling should simply be unavailable inside the source/track/sink model. The "energy" concept (although I'd probably have used "phlogiston" instead, just to be cute) is nice, but seems like a lot of mental model for this special case. Sinks can do what sinks want to do, but the track configurations won't do it for them.

I could probably be OK with removing upscaling.  I didn't go there
because I didn't want to add too much at once.  It's also something
that might be hard to achieve for some cameras.

> As for making it simpler.... I don't like being trapped into only mandatory constraints, any more than I like being trapped into only optional ones (as I've argued with Rich before). The expressive power of having both seems worthwhile to me. I may be strange in my head.

The only value optional constraints provide is the ability to prefer
one source over another.  Personally, I find the idea that an
application is picking sources based on some inherent technical
quality to be dubious.  More likely, it is the user who will be
picking sources and that will be based on what it is that the source
is capturing (i.e., where the camera is pointed is more important than
how many pixels it can pump).

We should set some precedence rules that lead to picking the "best"
source in a predictable fashion, e.g., pick the camera with the most
pixels.  Then we get consistent behaviour, after which I see very
little value in optional constraints.

I also see few cases where mandatory constraints are useful for the
above reason (I largely agree with Rich).  In most cases, mandatory
constraints are a liability and will cause unnecessary failures.
Where I do see them as useful is in selecting facing appropriate to
the application (user/environment) and for selecting the last camera
that was used for this application (sourceId).  It will be rare cases
where other constraints are sensible choices.  We should be very
careful to warn developers of what the dangers are.

Anyhow, I digress.  Optional constraints are only useful for initial
selection.  Optional constraints are complicated.  Some of that
functionality can be gained by using settings.  They can be added
(restored) later if someone has a super-interesting use case.
Otherwise, I'm all for removing them.

On 15 February 2013 05:51, Stefan Håkansson LK
<stefan.lk.hakansson@ericsson.com> wrote:
> * Is there really any point in having other than "min" constraints for
> energy? I mean, you'd want a minimum resolution, frame-rate, ... but would
> what would the gain of setting a "max" be? (OTOH, max would be important in
> a network transport situation in order to not send bits that would have no
> value for the consumer at the other end of a PeerConnection - but this
> should be dealt with in the webrt WG)

As described, you can set a max for energy, but that has no effect on
source selection, it only applies to the output of the track.

If you want to cap resolution, that doesn't guarantee that the camera
is going to produce fewer pixels.  That way, other users of the camera
can get the higher pixel count; you get a downscaled value.


> * As far mandatory and optional constraints go, I would be fine in removing
> the optional ones, but then we should standardize a way for (certain)
> consumer settings to travel upstream (up-track?) over a PeerConnection - and
> yes, that part is for another WG. I am primarily thinking about video width
> and height. Even in the case the app has no wish to use mandatory
> constraints for this the source should adapt to the display surface of the
> video element.

This is something that I'd be interested in being able to do, but we
should leave that to the WebRTC folks to decide.

> * Could not also an "energy" constraint/setting lead to an overconstrained
> situation? If you put min framerate to 40, while the camera can produce a
> maximum of 30.

Certainly.  If you set a selection criterion that can't be met by any
source, then you don't get a source.  Minimum values do reduce the
number of sources (and operating modes) you have available to select
from.  Set the minimum too high and the set of sources that can meet
that minimum will be reduced, and the minimum set size is zero.

> * (This is not related to your mail, but a reflection) Most constraints seem
> to be applicable both for source selection and for applying settings to
> already selected sources. But, for source selection the by far most usable
> constraint seem to be the direction of the camera (front/rear) - but this
> is(?) unusable in the settings as the camera can't be moved from the front
> to the rear of the device.

Travis made the same point for sourceId.  Obviously, once you attach a
source to a track, you can't switch sources, so that limits what
settings you can apply.

> * I share your fear regarding complexity. It is not only that we need to
> spec this up in a way that can be interpreted in only one way and that
> implementers have to implement it all - we will also build a huge space for
> testing where we need to verify that different implementations behave the
> same (or at least similar)

I prefer to avoid doing that sort of work entirely.  The best outcome
is a well-specified standard with a minimal, but complete set of test
cases.  Lower complexity means easier to implement, easier to test,
easier to specify correctly.  I don't see a downside, unless an
important use case gets missed.  Now that I looked at it, I'm not
convinced that we have a strong enough case for optional constraints.
Sure I can concoct a use case, but I'm not finding it compelling
enough to warrant the complexity.  Any use case that is sufficiently
compelling uses mandatory constraints, not optional ones.

Received on Friday, 15 February 2013 17:59:05 UTC