Re: [mediacapture-main] How to implement web-compatible camera downscaling?

> FWIW, I agree with @burnburn that getUserMedia is not a post-processing API.

So we should remove echoCancellation, autoGainControl and noiseSupression constraints because they are post-processing?

> What's the more likely reason for that popularity?
>
> 1. People wanted Chrome to follow FF + Edge in implementing ideal as the spec intended, or
> 2.  They wanted min-max ranges around their downscaled resolutions, because ... screen-sharing?

I can only read what people write in their bugs and what's written in the spec, not what's in other people's minds. Most bugs filed after the spec-compliant processing of video constraints shipped in Chrome 59 were about the behavior of screen sharing changing due to different default behavior with the new algorithm. Those bugs have been fixed since (while still conforming to the spec)  and I have not seen any other significant bugs about video-constraint processing.
More specifically, I have not seen evidence to suggest that the two reasons you present were what made the bugs popular, but, like I said, I can't read people's minds.

> The main complaint we hear is developers expecting getUserMedia to produce exactly the dimensions they specify "because it works in Chrome". It's a completely different model whether one scales at the source or the sink.

That sounds to me more like those users are OK with Chrome's implementation of the spec and not that they want Chrome to change its current implementation to match Firefox and Edge in this regard. Otherwise I think they would be filing Chrome bugs.
Also, I have seen Chrome users asking for video mirroring, which sounds at least reasonable to me since the spec has other post-processing constraints.

> I think the spec works well enough here, actually. Drivers can offer divide-by-two, divide-by-three etc. decimated frame-rates with few side effects, whereas apps wouldn't know which ones are good, and they're close enough for most app use. I think it would be a mistake to give continuous frame-rate control to the app.

Chrome already gives apps continuous frame-rate control and I have not seen users complain about it yet. I did see Chrome bugs filed when frame-rate control was not working correctly.

> These divide-by rates can be added by the UA as well if the driver doesn't offer them. So "native" doesn't have to mean 100% native, just discrete (good) choices rather than a continuous value space devoid of hints.

I agree that it's a good idea to allow applications to express that they prefer or require "native" values, even if no Chrome users have filed bugs asking for that feature. Our disagreement is about how to introduce that concept in the spec.

My opinion is that additional constrainable properties are a better fit for this than overloading the meaning of max/exact/ideal.

The constraints algorithm (as written in the spec) basically consists in:
1. Start with a set of valid candidates (each having a specific value for each constrainable property)
2. Discard the ones that do not satisfy the constraints specified by the app using min/max/exact.
3. Use ideal as the primary criterion to choose among the remaining candidates (ties among those that equally satisfy ideal are solved in an implementation specific manner). 

The nice thing about this approach is that it is generic enough that it applies equally to audio, video and potentially many other track types. Similarly, it is generic enough to allow many different types of constrainable properties (including post-processing ones, even if you think getUserMedia is not a post-processing API) without requiring special cases or exceptions for any of them. New filtering criteria or new types of candidates can be introduced with the addition of new constrainable properties with relatively low impact on the spec, existing applications and, possibly, implementations. 
My argument is that what we want to do in practice is add a new filtering type to Chrome and new candidate types (and the corresponding filtering) to Firefox and Edge. I think this is a good match for new constrainable properties.

My arguments against the overloading of min/max/exact/ideal are:
1. It adds nonintuitive meaning to those keywords, which may create user confusion.

2. It does not allow constrained non-native tracks. To overcome the first part you have to start adding exceptions (see 5).

3. It does not allow expressing a preference (without hard requirement) for native tracks. Wrt the second part, I guess you can change the definition to mean that min/max/exact means "prefer" but no hard requirement, in which case you need to make the spec even more complex by adding the concept of concurrency and explaining what is permitted under concurrency and what not.

4. It breaks existing applications. For example, many existing applications that use {echoCancellation: {exact: true}} or {echoCancellation: {ideal: true}} expect software echo cancellation (i.e., post-processing). If exact/ideal are redefined to also mean "native" or "no post-processing", then these applications would break unless we start adding exceptions for certain properties and/or certain types of tracks, making the algorithm not generic anymore. 

5. Once we start adding these specific exceptions we run the risk of creating more problems down the road. For example, it did not take long for you to propose an exception for min because you realized that users might want to constraint the minimum value of width or height of a rescaled resolution. What about constraining the maximum aspect ratio? What if we want to use audio resampling to implement the sampleRate constraint and want to support both downsampling and upsampling? Does that mean that we also need to introduce an exception for max and exact? Does that mean that the only overloaded keyword should be ideal, or that the exceptions would be different depending on the constraint and track type?
Honestly, I think this has the potential of making discrepancies of browser behavior worse due to bugs and different interpretations of the exceptions, especially when it comes to correlated properties (different browsers might interpret different sets of properties as correlated) and different treatment of concurrent and non-concurrent scenarios.

So far, the arguments you have brought  against introducing new constrainable properties are:
1. The original intention of the authors of the spec was to mandate Firefox-like behavior (native-mode disovery). My answer to that is that this is irrelevant now because this intention is not expressed in any way whatsoever in the part of the spec that defines the constrainable pattern. The question now is how to introduce this intention in the spec with minimal disruption to existing applications, implementations and the spec itself, given there is already a considerable investment in the spec as currently written by both application developers and implementation writers. 
2. The proposed constraints introduce post-processing and getUserMedia is not a post-processing spec. My answer is that, as written, getUserMedia is a post-processing spec. There is language in the spec that says "User Agent processing, such as downsampling, may be used to ensure that all tracks have appropriate media." Also, there are constrainable properties that explicitly refer to post-processing techniques, such as echoCancellation, autoGainControl and noiseSuppression.

Your arguments in favor of overloading min/max/exact/ideal:
1. It matches the original intention of the spec authors. Like I said, I find this irrelevant since that intention is neither explicitly nor implicitly expressed in the spec. I would argue that the genericness of the spec wrt constraints suggests to me that another intention of the authors was to keep it generic and free from exceptions and concepts specific to particular track types or properties. This opinion is based on what's actually written in the spec.
2. The proposed changes are a SHOULD. I think this is a disadvantage. This would mean that some browsers might opt to fully implement this SHOULD, others might implement it in some circumstances (e.g., only when there is no concurrent access), while others might not implement it at all. This could result in a even more divergent browser behavior together with a more complex spec, potentially making the problem worse.

My conclusion is that the case for using new constrainable properties to control rescaling is stronger than giving extra meaning to min/max/exact/ideal because the former has much lower compatibility and interoperability risks. Moreover, I think that leaving the spec unmodified and letting the browser choose how to do rescaling is also better than redefining min/max/exact/ideal. Nothing in the current spec prevents Firefox from, for example, maintaining its current behavior in the non-concurrent case and resorting to rescaling only when concurrency and constraints prevent native mode for all tracks.

I apologize if I misunderstood or forgot to include some of your arguments, so please feel free to correct me and add new thoughts so that we can advance the discussion.


-- 
GitHub Notification of comment by guidou
Please view or discuss this issue at https://github.com/w3c/mediacapture-main/issues/472#issuecomment-320953835 using your GitHub account

Received on Tuesday, 8 August 2017 13:21:00 UTC