Re: Proposal: Constraints as dictionaries

TL;DR: search for "Interesting".

On 11/21/13 3:33 AM, Adam Bergkvist wrote:
> Thanks, Jan-Ivar, for putting this proposal together. I wonder how 
> much of your brain capacity that has been consumed by constraints 
> lately. :)

It's safe to say my brain is over-constrained. :-)

> On 2013-11-20 22:30, Jan-Ivar Bruaroey wrote:
>>       A. Don't violate WebIDL
>>
>>
>> Constraints must be WebIDL dictionaries. We cannot afford to reinvent
>> dictionaries or webidl just for Constraints, at this point. The WebIDL
>> WG have thought more about JS and JS processing models than I think we
>> have, and for the complexity and long-tail use-cases inherent in this
>> API, we need explicit definitive language that implementers can follow,
>> and WebIDL is that language.
>>
>> WebIDL dictionaries ignore unknown keys. The following tweaks in our
>> model let constraints be dictionaries:
>>
>>   1. Conservative programmer calls if
>> (!getGumKnownConstraints().hasOwnProperty("3D")) then bail
>>   2. getUserMedia({ mandatory: { 3D: true } }, succ, fail); // where 3D
>> is ignored if unimplemented
>>   3. (minor) Remove limit of one key-value pair in dictionaries in
>> optional array (over-specified)
>>   4. KeepConstrainable as a pattern, but root it in actual gUM types
>> (remove abstractions, 'any' etc.)
>>   5. Change abstract PropertyValueRange to UnsignedLongRange, FloatRange
>> etc.
>>
>> Most controversial difference: The API for detecting browser-support for
>> a constraint is now explicit, and gUM() itself only constrains by
>> properties it knows.
>>
>> This avoids blanket blocking by browser for bad as well as good cameras,
>> something simple apps may not have signed on for, and lets apps
>> explicitly block access on those same browsers with one line of code.
>>
>> Not new: Code that plans to control the camera must already use
>> getCapabilities() to learn the known things they can modify, so there is
>> no added cost in those cases.
>>
>> We then describe Constraints, Capabilities and Settings in the spec
>> using WebIDL instead of abstract prose. I circulated
>> http://lists.w3.org/Archives/Public/public-media-capture/2013Nov/0084.htmlearlier 
>>
>> which shows the proposed WebIDL at the bottom. Using WebIDL
>> descriptively in this manner does not preclude IANA.
>
> I think what you say here makes sense.

Great!

> Is the thinking that two optional constraints with the same priority 
> (point 3.) should be evaluated together as "constraint 1" && 
> "constraint 2"?

Yes. This falls out for free in any implementation that figures out it 
can reuse the same SatisfyConstraint() sub-function for mandatory and 
optional (array of  single-key dictionaries). Treating the 
key-value-pair as a dictionary also answers what its JS processing model 
is, something the spec leaves out today. I consider this part minor 
however, since testing Object.keys(pair).length == 1 would be trivial, 
if the spec were to says what should happen.

>>       B. Don't leak
>>
>>
>> Stop returning ConstraintNotSatisfiedError, because it leaks information
>> about capabilities to the website without user consent, letting a
>> malicious website paint a full picture after a few visits.
>>
>> Instead, always bring up the permission prompt, even when there is no
>> match.The UA MUST warn the user differently of such non-matches and
>> offer the Deny-equivalent choice only. The UA MUST NOT let the user
>> grant access to a camera in this case (not because of a leak, but to
>> give webpages an invariant for known properties).
>
> We said on the TPAC meeting that we should make mandatory constraints 
> "harder to use" since they are so easy to miss-use. However, the 
> conclusion I come to when looking at our spec is that mandatory 
> constraints are much easier to understand and use (compared to 
> optional ones). With mandatory you get what you ask for (with some 
> limitations), and if the browser no longer can sustain a mandatory 
> constraint, you will be notified. The point of providing optional 
> constraints is less obvious. They may be disregarded at any time and 
> to UA won't tell you when doing so. Optional constraints do provide a 
> feature to prefer high frame rate over resolution, but it's not that 
> obvious how it's done. I might be missing something but will anyone 
> even use optional constrains under these circumstances?

I hope so! To me, how they work in concert is more important than how 
they work individually. I agree the interaction is confusing, which is 
why I proposed the new syntax.

> This proposal doesn't make mandatory constraints harder to use, but it 
> could make them a bit less exclusive.
>
> With this proposal, even if you don't have a good enough device, you 
> will at least get something, the prompt, even if the app has an empty 
> error callback on getUserMedia() that ideally should have informed the 
> user about the situation. It's now up to the browser to let users 
> down, but at least it can be done consistently.
>
> We've have, pretty much since day one, said that it should be possible 
> to provide a file as a media source to not exclude users. (The 
> conclusion at some discussion was that picking a file would be similar 
> to connecting a new device.) By always prompting the user we have a 
> better opportunity to offer that service, compared with the 
> ConstraintNotSatisfiedError approach. This might be a crazy idea, but 
> we could even offer a fake device that meets the requirements of the 
> application.
>
>>       C. New syntax
>>
>>
>> A new syntax for Constraints only, is aimed at moving us away from the
>> mandatory/optional language to a model focused on expressiveness, and
>> away from Constraints being "things" to being dictionaries used in a
>> pattern, just like Settings and Capabilities are now.
>>
>> In this syntax, Constraints is an array of Settings (dictionaries) in
>> decreasing order of preference, where a set is accepted only if all its
>> specified keys are. The first accepted set is chosen.
>>
>> To describe your camera preferences in fine detail, you provide all the
>> combinations of settings you accept, specifying only the things you care
>> about, in all the combinations you accept.
>>
>> I show a width/height example here
>> http://lists.w3.org/Archives/Public/public-media-capture/2013Nov/0052.htmlwhich 
>>
>> in turn has a link to the original post.
>>
>> Let me elaborate on the example in the original post (sans typo):
>>> Consider an alternative to:
>>>           { mandatory: { a, b }, optional: [ { c }, { d } ] }
>>> like this:
>>>         [ { a, b, c, d }, { a, b, c }, { a, b, d }, { a, b } ]
>>> Old familiars and what they look like now:
>>>      { mandatory: { a, b } } ==> [ { a, b } ]
>>> and separately:
>>>      { optional: [{ a }, { b }] } ==> [ { a }, { b }, {} ]
>>
>> Benefits of new syntax:
>> + Simpler. No keyword-dependent structure variance ([{}] vs. {}).
>> + Advantage of being able to combine related things like width + height
>> naturally and logically.
>> + {} at the end of the array naturally makes everything optional, since
>> it means "I'll accept anything".
>> + More power (allows for finer, more explicit selection).
>> + (AND)OR array logic is easier to read than the current optional-array
>> logic (esp. width/height combos)
>> + Bye bye misunderstood mandatory keyword. Hello dictionaries. <3
>>
>> Cons:
>> - Can't handle large number of optionals, grows exponentially in
>> edge-cases. (6 = ugh, 30 = no way)
>>    I'm banking on there always being patterns in what programmers desire
>> (OR-ing sets not 1by1. Try it!)
>> - Pilot error: Forget ending array with {} and everything is mandatory.
>>
>> Since we no longer tailor any data-types for our special needs, and rely
>> solely on well-known constructs, how things work and don't work should
>> be much clearer both to developers and programmers.
>
> This looks reasonable, at least at first glance. :)

Great!

> It's also a better fit with Constrainable.applyConstraints() since 
> optional constraints don't really make sense in that case. I'm 
> imagining feeding applyConstraints() with the same type as the 
> elements in the array given to getUserMedia().

Interesting. To be clear, I wasn't proposing that, but I'm glad the 
decoupling opens up the discussion. Feeding it the array itself would 
match how it works today.

You have a point. Constraints let you be imprecise, saying things like 
"give me something like 40 - 50 fps, but no lower than 1024x768". This 
abstraction seems like overkill when you can read the capabilities of 
the device.

Just to be sure: track. applyConstraints() can never pick a different 
camera, right? That seems obvious inside a stream with other tracks, but 
it's never spelled out. If it can't then I don't see why we need 
constraints here, setSettings() would suffice.

> We should really get started on the document how constrains solves 
> different use-cases. Obvious limitations will show up when the method 
> face a real life situation.

I agree.

> /Adam

.: Jan-Ivar :.