- From: Jan-Ivar Bruaroey <jib@mozilla.com>
- Date: Thu, 21 Nov 2013 15:49:53 -0500
- To: Adam Bergkvist <adam.bergkvist@ericsson.com>, "public-media-capture@w3.org" <public-media-capture@w3.org>
- Message-ID: <528E71F1.2050007@mozilla.com>
TL;DR: search for "Interesting". On 11/21/13 3:33 AM, Adam Bergkvist wrote: > Thanks, Jan-Ivar, for putting this proposal together. I wonder how > much of your brain capacity that has been consumed by constraints > lately. :) It's safe to say my brain is over-constrained. :-) > On 2013-11-20 22:30, Jan-Ivar Bruaroey wrote: >> A. Don't violate WebIDL >> >> >> Constraints must be WebIDL dictionaries. We cannot afford to reinvent >> dictionaries or webidl just for Constraints, at this point. The WebIDL >> WG have thought more about JS and JS processing models than I think we >> have, and for the complexity and long-tail use-cases inherent in this >> API, we need explicit definitive language that implementers can follow, >> and WebIDL is that language. >> >> WebIDL dictionaries ignore unknown keys. The following tweaks in our >> model let constraints be dictionaries: >> >> 1. Conservative programmer calls if >> (!getGumKnownConstraints().hasOwnProperty("3D")) then bail >> 2. getUserMedia({ mandatory: { 3D: true } }, succ, fail); // where 3D >> is ignored if unimplemented >> 3. (minor) Remove limit of one key-value pair in dictionaries in >> optional array (over-specified) >> 4. KeepConstrainable as a pattern, but root it in actual gUM types >> (remove abstractions, 'any' etc.) >> 5. Change abstract PropertyValueRange to UnsignedLongRange, FloatRange >> etc. >> >> Most controversial difference: The API for detecting browser-support for >> a constraint is now explicit, and gUM() itself only constrains by >> properties it knows. >> >> This avoids blanket blocking by browser for bad as well as good cameras, >> something simple apps may not have signed on for, and lets apps >> explicitly block access on those same browsers with one line of code. >> >> Not new: Code that plans to control the camera must already use >> getCapabilities() to learn the known things they can modify, so there is >> no added cost in those cases. >> >> We then describe Constraints, Capabilities and Settings in the spec >> using WebIDL instead of abstract prose. I circulated >> http://lists.w3.org/Archives/Public/public-media-capture/2013Nov/0084.htmlearlier >> >> which shows the proposed WebIDL at the bottom. Using WebIDL >> descriptively in this manner does not preclude IANA. > > I think what you say here makes sense. Great! > Is the thinking that two optional constraints with the same priority > (point 3.) should be evaluated together as "constraint 1" && > "constraint 2"? Yes. This falls out for free in any implementation that figures out it can reuse the same SatisfyConstraint() sub-function for mandatory and optional (array of single-key dictionaries). Treating the key-value-pair as a dictionary also answers what its JS processing model is, something the spec leaves out today. I consider this part minor however, since testing Object.keys(pair).length == 1 would be trivial, if the spec were to says what should happen. >> B. Don't leak >> >> >> Stop returning ConstraintNotSatisfiedError, because it leaks information >> about capabilities to the website without user consent, letting a >> malicious website paint a full picture after a few visits. >> >> Instead, always bring up the permission prompt, even when there is no >> match.The UA MUST warn the user differently of such non-matches and >> offer the Deny-equivalent choice only. The UA MUST NOT let the user >> grant access to a camera in this case (not because of a leak, but to >> give webpages an invariant for known properties). > > We said on the TPAC meeting that we should make mandatory constraints > "harder to use" since they are so easy to miss-use. However, the > conclusion I come to when looking at our spec is that mandatory > constraints are much easier to understand and use (compared to > optional ones). With mandatory you get what you ask for (with some > limitations), and if the browser no longer can sustain a mandatory > constraint, you will be notified. The point of providing optional > constraints is less obvious. They may be disregarded at any time and > to UA won't tell you when doing so. Optional constraints do provide a > feature to prefer high frame rate over resolution, but it's not that > obvious how it's done. I might be missing something but will anyone > even use optional constrains under these circumstances? I hope so! To me, how they work in concert is more important than how they work individually. I agree the interaction is confusing, which is why I proposed the new syntax. > This proposal doesn't make mandatory constraints harder to use, but it > could make them a bit less exclusive. > > With this proposal, even if you don't have a good enough device, you > will at least get something, the prompt, even if the app has an empty > error callback on getUserMedia() that ideally should have informed the > user about the situation. It's now up to the browser to let users > down, but at least it can be done consistently. > > We've have, pretty much since day one, said that it should be possible > to provide a file as a media source to not exclude users. (The > conclusion at some discussion was that picking a file would be similar > to connecting a new device.) By always prompting the user we have a > better opportunity to offer that service, compared with the > ConstraintNotSatisfiedError approach. This might be a crazy idea, but > we could even offer a fake device that meets the requirements of the > application. > >> C. New syntax >> >> >> A new syntax for Constraints only, is aimed at moving us away from the >> mandatory/optional language to a model focused on expressiveness, and >> away from Constraints being "things" to being dictionaries used in a >> pattern, just like Settings and Capabilities are now. >> >> In this syntax, Constraints is an array of Settings (dictionaries) in >> decreasing order of preference, where a set is accepted only if all its >> specified keys are. The first accepted set is chosen. >> >> To describe your camera preferences in fine detail, you provide all the >> combinations of settings you accept, specifying only the things you care >> about, in all the combinations you accept. >> >> I show a width/height example here >> http://lists.w3.org/Archives/Public/public-media-capture/2013Nov/0052.htmlwhich >> >> in turn has a link to the original post. >> >> Let me elaborate on the example in the original post (sans typo): >>> Consider an alternative to: >>> { mandatory: { a, b }, optional: [ { c }, { d } ] } >>> like this: >>> [ { a, b, c, d }, { a, b, c }, { a, b, d }, { a, b } ] >>> Old familiars and what they look like now: >>> { mandatory: { a, b } } ==> [ { a, b } ] >>> and separately: >>> { optional: [{ a }, { b }] } ==> [ { a }, { b }, {} ] >> >> Benefits of new syntax: >> + Simpler. No keyword-dependent structure variance ([{}] vs. {}). >> + Advantage of being able to combine related things like width + height >> naturally and logically. >> + {} at the end of the array naturally makes everything optional, since >> it means "I'll accept anything". >> + More power (allows for finer, more explicit selection). >> + (AND)OR array logic is easier to read than the current optional-array >> logic (esp. width/height combos) >> + Bye bye misunderstood mandatory keyword. Hello dictionaries. <3 >> >> Cons: >> - Can't handle large number of optionals, grows exponentially in >> edge-cases. (6 = ugh, 30 = no way) >> I'm banking on there always being patterns in what programmers desire >> (OR-ing sets not 1by1. Try it!) >> - Pilot error: Forget ending array with {} and everything is mandatory. >> >> Since we no longer tailor any data-types for our special needs, and rely >> solely on well-known constructs, how things work and don't work should >> be much clearer both to developers and programmers. > > This looks reasonable, at least at first glance. :) Great! > It's also a better fit with Constrainable.applyConstraints() since > optional constraints don't really make sense in that case. I'm > imagining feeding applyConstraints() with the same type as the > elements in the array given to getUserMedia(). Interesting. To be clear, I wasn't proposing that, but I'm glad the decoupling opens up the discussion. Feeding it the array itself would match how it works today. You have a point. Constraints let you be imprecise, saying things like "give me something like 40 - 50 fps, but no lower than 1024x768". This abstraction seems like overkill when you can read the capabilities of the device. Just to be sure: track. applyConstraints() can never pick a different camera, right? That seems obvious inside a stream with other tracks, but it's never spelled out. If it can't then I don't see why we need constraints here, setSettings() would suffice. > We should really get started on the document how constrains solves > different use-cases. Obvious limitations will show up when the method > face a real life situation. I agree. > /Adam .: Jan-Ivar :.
Received on Thursday, 21 November 2013 20:50:26 UTC