Responsive constraints (was: Re: Mandatory vs optional constraints)

Harald Alvestrand wrote:
> Changing the subject line, since this has absolutely nothing to do with
> the syntax or semantics of the resolution constraints....

I believe this is a discussion on the merits of either having a 
responsive approach to constraint management vs. specifying up-front 
constraints at web application start-up.

That really boils down to a question of user experience when using 
getUserMedia and is not a decision between mandatory vs optional 
constraints. Both are still supported either way.

>
> On 08/27/2012 11:54 AM, Rich Tibbett wrote:
>> Let's take a look at the example you provide in your draft below:
>>
>> getUserMedia({
>> video: { mandatory: { minWidth: 1024, minHeight: 768 } }
>> }, successCallback, errorCallback);
>>
>> Let's assume for a moment that this _may_ work only small proportion
>> of the time. Once we start applying more constraints that number
>> continues to fall exponentially.
>>
>> What the above code is suggesting is to support an exclusive design
>> approach rather than an inclusive design approach - where if a user
>> doesn't meet developer-decided expectations they will be flat out
>> rejected without further explanation or regard.
> Rich, you and I seem to have a strong non-connect here.
>
> I think developers are interested in building an experience that is good
> for their users. It's our business to make sure that they have the tools
> to do so.

We should also be encouraging ubiquitous usage of services and 
incentivizing developers to take responsibility for supporting as many 
web users as they can.

>>
>> My question on your code above is: what if you have successfully
>> obtained the HD video above and then you realize that the user is
>> actually on a really slow network that would not be capable of
>> streaming this HD video?
>>
>> You're going to want to scale that video down to something more
>> amenable to the current internet connection.
> That depends.
>
> If I as an application developer know, positively, that this particular
> camera is going to be used for projecting an image of an A4 page of
> text, and that the recipient of the stream expects to READ that text, I
> indeed know that something less than 1024x768 is useless for their purpose.
>
> The correct behaviour if this can't be done is to put up an error box
> saying "Sorry, I can't do that kind of projection with this camera".

But is that really the behavior you want as a user? Essentially, this 
web application is poorly designed. Please see below.

>
> It's my business as an app developer to do that only when it's
> reasonable *in the context of that application* to do so.
>
>> What I'd like to therefore see instead is something similar to the
>> following:
>>
>> getUserMedia({ video: true }
>> }, successCallback, errorCallback);
>>
>> function successCallback(stream) {
>> // attempt to scale up to HD resolution (if it is supported
>> // on this device)
>> stream.videoTracks[0].width = 1024;
>> stream.videoTracks[0].height = 768;
> This is, of course, totally user-hostile.
>
> If the user has just bought a 4k camera, and his browser, his computer
> and his network all support that, he has a reasonable expectation that
> he should just plug it in in place of his old camera, and it should Just
> Work - because the app has (correctly) told the browser that there are
> *minimum* requirements, but no *maximum* requirements.
>
> Your code example, if approached in this fashion, is actively hurting
> the future-proofing of the Web.

You may have to explain this statement as I don't quite follow.

In your example, the old camera was flat out rejected up-front without 
much ado. In the responsive approach the old camera worked, albeit at a 
lower resolution, because the developer was incentivized to make it work 
to improve the user experience of their web application (because 
otherwise the user just got a 'Can't do' message). The new camera just 
happened to provide higher resolution than the old camera but still 
worked in the same way.

>>
>> // setup and start streaming the video (out of scope)
>>
>> }
>>
>> During the course of streaming, if you want to subsequently downgrade
>> the pixel density of the video, at any time, to account for e.g.
>> varying connection speeds or e.g. screen-in-screen display, you should
>> be able to do something akin to the following:
>>
>> stream.videoTracks[0].width *= .5;
>> stream.videoTracks[0].height *= .5;
>
> This is also unrealistic, for another reason; the application has no way
> of knowing what numbers to aim for. If the camera (as is typical)
> supports resolutions only in steps of 16 pixels, multiplying the present
> resolution with a floating point number may succeed and it may fail; if
> it fails, you have to have rules that determine what the actual value
> set is: Unchanged, round up, round down, round closest, or
> "implementation decides".
>
> Again, the right approach is to specify the constraints that the
> application knows and the browser cannot know, and let the browser work
> it out.

The browser is still involved in capability changes to media stream 
tracks every step of the way in the responsive constraints proposal.

>>
>> The great thing about this approach is, as Adam pointed out on the
>> recent conference call, that the web page can report in detail why a
>> web application is not willing to work with a camera media stream
>> provided to that web page. Even though the web browser provided a
>> camera stream to the web app as requested the application can decide
>> not to work with it if they choose and they can say ('You browser
>> doesn't support HD video' or 'You're web camera is not capable of 30
>> fps', etc).
>>
>> So for example, if a web application only supports HD video, which I
>> would argue is a _very poor_ design approach for something like the
>> web, a developer could enforce that behavior in their web application
>> with something similar to the following code:
>>
>> function successCallback(stream) {
>> // attempt to scale up to HD resolution (if it is supported
>> // on this device)
>> stream.videoTracks[0].width = 1024;
>> stream.videoTracks[0].height = 768;
>>
>> // this application ONLY supports HD video. Abort if we can't
>> // modify the current stream to HD resolution
>> if(stream.videoTracks[0].width !== 1024 ||
>> stream.videoTracks[0].height !== 768) {
>> console.log("You must have a HD Camera to use this app. Aborting.");
>> return;
>> }
>>
>> // setup and start streaming the video (out of scope)
>>
>> }
>>
>> Of course, what a conscientious web developer would do is implement
>> the first snippet which I'll reproduce here:
>>
>> function successCallback(stream) {
>> // attempt to scale up to HD resolution (if it is supported
>> // on this device)
>> stream.videoTracks[0].width = 1024;
>> stream.videoTracks[0].height = 768;
>>
>> // setup and start streaming the video (out of scope)
>>
>> }
>>
>> ...which says: I _really_ want to stream HD video if that's a
>> supported option. _Otherwise, I'll fallback to whatever resolution the
>> web cam supports_.
> Which you can do, by specifying the constraint as optional.
>
> What you are arguing is that you want to take the expressive power away
> from the application programmer to tell the browser which of his
> constraints he must satisfy in order to have an useful experience. You
> then force the browser to perform work (allocating cameras, running
> through user permission dialogs) in order to let the user access the
> decision message saying "Can't do".
>
> Why is that an advantage to anyone?

For me this is the crux of the discussion. It's one of user experience.

Let's imagine for a minute that this is the way it is implemented. That 
the user will receive a prompt to share their camera and your web 
application will then decide that it can't do whatever it intends to do 
at the provided resolution, because it has been designed for only one 
demographic of web users. It thereby notifies the user and then flat-out 
rejects them any further use of your web application.

That would, as you've suggested, be a bad user experience.

The web application has not been designed to account for the 
heterogeneous nature of web cameras, the web and its users.

What a developer would be inclined to do to improve that user experience 
is to tweak their web application to provide a similar experience at 
e.g. a lower resolution than originally anticipated/tested in their 
limited development environment. Perhaps it would function in the same 
way as something of a higher resolution but above the resulting 
displayed video stream it notes that the web application really works 
best at higher resolutions but since those are not available, we'll 
provide service on a best-effort basis.

The developer could perhaps choose different media encodings for better 
visual rendering depending on the content of the stream (slides vs. 
moving pictures).

The developer could perhaps block off some of the more advanced 
functionality but still provide some basic level of service to the user 
to stream their video.

The developer is self-incentived to at least try to make their 
application support the majority of web users because it means less 
denial of service and therefore less bad experiences by web users in 
general. This model encourages better design decisions thereby having 
the net effect of allowing (many) more people to use that application on 
the web than it was originally designed for. Perhaps that experience is 
not at the same level of service as ideal hardware could provide but it 
works at some basic level.

Now users don't get denied service up-front nor do they get rejected 
from using the web application once a camera stream has been provided, 
albeit of lesser quality. They get to use the web application but the 
user can expect some things not to be available or some things to be 
blocked off to users that don't have ideal hardware.

That would be a good user experience based on responsive web design 
principles.

That would be a much improved, much more inclusive user experience than 
would be provided in the current proposal because it incentives 
developers to support more web users (or it doesn't and the web 
application is 'penalized' for being too exclusive).

We see this type of service all the time. Video is often adaptive based 
on the bandwidth available and the same principle is in play here. Not a 
single solid use case has been put forward to explain why a web 
application will only take HD video or nothing.

>
>>
>> As we've seen above, a web application can enforce any particular
>> capabilities on to a web cam stream to be mandatory but that occurs
>> after getUserMedia invocation has already successfully returned a web
>> camera stream. The implication here is that we can support all of the
>> available use cases. The only difference seems to be one of user
>> experience. If a web application only supports HD video resolutions
>> and refuses to fallback to any stream of lesser quality then that is
>> to be considered a failing of the developer/service.
>>
>> The side effect of this approach is to make web developers fully aware
>> that if they do not intend to work within the capabilities of the
>> majority (read: 99%+) of their user's hardware then there will be
>> penalties for doing so in the form of having to inform their users
>> that they're not willing to work with any provided camera streams
>> because they don't meet some pre-determined, idealized capabilities
>> the initial developers had in mind.
>>
>> I'd like to see our efforts moving in this direction away from the
>> intentional exclusion of web users with lesser hardware capabilities
>> such as those emerging slowly on to the web from e.g. the developing
>> world or those with lesser hardware capabilities or those that can't
>> or don't keep up with having the latest and greatest gadgets all of
>> the time.
>>
>> Both inclusive and exclusive types of services remain implementable in
>> this approach. Ultimately it is up to developers to decide which
>> category they want to be in and implement their apps/services
>> accordingly. Market forces will then decide whether those services
>> succeed or not.
> This is true when you let the browser help you do the selection, and is
> true when you leave the selection up to the Javascript. Reducing the
> expressive power of constraints does not help you reach your goal.
>>
>> Web applications that are adaptive and have flexibility built-in to
>> their streaming capabilities should win. Exclusionary web
>> apps/services that require 'ideal' user hardware and remain inflexible
>> to changing user hardware trends and environments should lose.
>
> Agreed. But reducing the expressive power of constraints is not (in my
> opinion) something that increases the chances of this happening.

Agreed. But giving developers up-front constraints as an 'easy-out' to 
avoid responsibility for providing responsive service to as many users 
as possible in a heterogeneous web environment is going to fragment and 
damage the long term health of the web.

Making the browser responsible for rejection of service up-front is also 
a convenient way to defer developer responsibility here. That would be bad.

- Rich

Received on Monday, 27 August 2012 12:48:16 UTC