Mandatory vs optional constraints (Re: Constraints for resolution: draft-alvestrand-constraints-resolution-00.txt

Changing the subject line, since this has absolutely nothing to do with 
the syntax or semantics of the resolution constraints....

On 08/27/2012 11:54 AM, Rich Tibbett wrote:
> Let's take a look at the example you provide in your draft below:
>
> getUserMedia({
>       video: { mandatory: { minWidth: 1024, minHeight: 768 } }
>    }, successCallback, errorCallback);
>
> Let's assume for a moment that this _may_ work only small proportion 
> of the time. Once we start applying more constraints that number 
> continues to fall exponentially.
>
> What the above code is suggesting is to support an exclusive design 
> approach rather than an inclusive design approach - where if a user 
> doesn't meet developer-decided expectations they will be flat out 
> rejected without further explanation or regard.
Rich, you and I seem to have a strong non-connect here.

I think developers are interested in building an experience that is good 
for their users. It's our business to make sure that they have the tools 
to do so.
>
> My question on your code above is: what if you have successfully 
> obtained the HD video above and then you realize that the user is 
> actually on a really slow network that would not be capable of 
> streaming this HD video?
>
> You're going to want to scale that video down to something more 
> amenable to the current internet connection.
That depends.

If I as an application developer know, positively, that this particular 
camera is going to be used for projecting an image of an A4 page of 
text, and that the recipient of the stream expects to READ that text, I 
indeed know that something less than 1024x768 is useless for their purpose.

The correct behaviour if this can't be done is to put up an error box 
saying "Sorry, I can't do that kind of projection with this camera".

It's my business as an app developer to do that only when it's 
reasonable *in the context of that application* to do so.

> What I'd like to therefore see instead is something similar to the 
> following:
>
> getUserMedia({ video: true }
>    }, successCallback, errorCallback);
>
> function successCallback(stream) {
>   // attempt to scale up to HD resolution (if it is supported
>   // on this device)
>   stream.videoTracks[0].width  = 1024;
>   stream.videoTracks[0].height = 768;
This is, of course, totally user-hostile.

If the user has just bought a 4k camera, and his browser, his computer 
and his network all support that, he has a reasonable expectation that 
he should just plug it in in place of his old camera, and it should Just 
Work - because the app has (correctly) told the browser that there are 
*minimum* requirements, but no *maximum* requirements.

Your code example, if approached in this fashion, is actively hurting 
the future-proofing of the Web.
>
>   // setup and start streaming the video (out of scope)
>
> }
>
> During the course of streaming, if you want to subsequently downgrade 
> the pixel density of the video, at any time, to account for e.g. 
> varying connection speeds or e.g. screen-in-screen display, you should 
> be able to do something akin to the following:
>
>   stream.videoTracks[0].width  *= .5;
>   stream.videoTracks[0].height *= .5;

This is also unrealistic, for another reason; the application has no way 
of knowing what numbers to aim for. If the camera (as is typical) 
supports resolutions only in steps of 16 pixels, multiplying the present 
resolution with a floating point number may succeed and it may fail; if 
it fails, you have to have rules that determine what the actual value 
set is: Unchanged, round up, round down, round closest, or 
"implementation decides".

Again, the right approach is to specify the constraints that the 
application knows and the browser cannot know, and let the browser work 
it out.
>
> The great thing about this approach is, as Adam pointed out on the 
> recent conference call, that the web page can report in detail why a 
> web application is not willing to work with a camera media stream 
> provided to that web page. Even though the web browser provided a 
> camera stream to the web app as requested the application can decide 
> not to work with it if they choose and they can say ('You browser 
> doesn't support HD video' or 'You're web camera is not capable of 30 
> fps', etc).
>
> So for example, if a web application only supports HD video, which I 
> would argue is a _very poor_ design approach for something like the 
> web, a developer could enforce that behavior in their web application 
> with something similar to the following code:
>
> function successCallback(stream) {
>   // attempt to scale up to HD resolution (if it is supported
>   // on this device)
>   stream.videoTracks[0].width  = 1024;
>   stream.videoTracks[0].height = 768;
>
>   // this application ONLY supports HD video. Abort if we can't
>   // modify the current stream to HD resolution
>   if(stream.videoTracks[0].width !== 1024 ||
>        stream.videoTracks[0].height !== 768) {
>     console.log("You must have a HD Camera to use this app. Aborting.");
>     return;
>   }
>
>   // setup and start streaming the video (out of scope)
>
> }
>
> Of course, what a conscientious web developer would do is implement 
> the first snippet which I'll reproduce here:
>
> function successCallback(stream) {
>   // attempt to scale up to HD resolution (if it is supported
>   // on this device)
>   stream.videoTracks[0].width  = 1024;
>   stream.videoTracks[0].height = 768;
>
>   // setup and start streaming the video (out of scope)
>
> }
>
> ...which says: I _really_ want to stream HD video if that's a 
> supported option. _Otherwise, I'll fallback to whatever resolution the 
> web cam supports_.
Which you can do, by specifying the constraint as optional.

What you are arguing is that you want to take the expressive power away 
from the application programmer to tell the browser which of his 
constraints he must satisfy in order to have an useful experience. You 
then force the browser to perform work (allocating cameras, running 
through user permission dialogs) in order to let the user access the 
decision message saying "Can't do".

Why is that an advantage to anyone?

>
> As we've seen above, a web application can enforce any particular 
> capabilities on to a web cam stream to be mandatory but that occurs 
> after getUserMedia invocation has already successfully returned a web 
> camera stream. The implication here is that we can support all of the 
> available use cases. The only difference seems to be one of user 
> experience. If a web application only supports HD video resolutions 
> and refuses to fallback to any stream of lesser quality then that is 
> to be considered a failing of the developer/service.
>
> The side effect of this approach is to make web developers fully aware 
> that if they do not intend to work within the capabilities of the 
> majority (read: 99%+) of their user's hardware then there will be 
> penalties for doing so in the form of having to inform their users 
> that they're not willing to work with any provided camera streams 
> because they don't meet some pre-determined, idealized capabilities 
> the initial developers had in mind.
>
> I'd like to see our efforts moving in this direction away from the 
> intentional exclusion of web users with lesser hardware capabilities 
> such as those emerging slowly on to the web from e.g. the developing 
> world or those with lesser hardware capabilities or those that can't 
> or don't keep up with having the latest and greatest gadgets all of 
> the time.
>
> Both inclusive and exclusive types of services remain implementable in 
> this approach. Ultimately it is up to developers to decide which 
> category they want to be in and implement their apps/services 
> accordingly. Market forces will then decide whether those services 
> succeed or not.
This is true when you let the browser help you do the selection, and is 
true when you leave the selection up to the Javascript. Reducing the 
expressive power of constraints does not help you reach your goal.
>
> Web applications that are adaptive and have flexibility built-in to 
> their streaming capabilities should win. Exclusionary web 
> apps/services that require 'ideal' user hardware and remain inflexible 
> to changing user hardware trends and environments should lose.

Agreed. But reducing the expressive power of constraints is not (in my 
opinion) something that increases the chances of this happening.

Received on Monday, 27 August 2012 10:26:32 UTC