Fwd: Capabilities API proposal

Since Harald forwarded Cullen's proposals, I thought it would be appropriate for me to forward the one I sent that started off the discussion in the webrtc list.  My proposal is at the end of this email.  I added the additional text immediately below to explain the reasoning that led to the proposal.

-- dan

Begin forwarded message:

> Resent-From: public-webrtc@w3.org
> From: Dan Burnett <dburnett@voxeo.com>
> Date: January 24, 2012 9:05:08 AM EST
> To: public-webrtc@w3.org
> Subject: Re: Capabilities API proposal
> archived-at: <http://www.w3.org/mid/9D8DFC63-C55C-4289-9543-FD1E09A22D16@voxeo.com>
> 
> I'm going to be replying to some other emails shortly, including Cullen's comments, but before I do I think it would be helpful for me to explain the thinking that led to my API proposal.
> 
> When we first began discussing this at Voxeo, as a thought experiment we began with the premise that a well-designed Hints API would make a Capabilities API unnecessary.  After all, if a web browser could sufficiently understand what an app author wanted to do, then there would be no need to query capabilities.
> We immediately ran into two problems with this.  First, this did not allow us to grey out or hide options known to be unavailable.  For example, if we knew that a camera capable of generating a decent resolution image was not available we would not want to even show that as an option in the UI.  Clearly this has privacy implications and should be discussed in the larger context of how much capability information should be available to the application (possibly differing based on whether permission has been given).  Second, we also realized that in many cases we had minimum requirements.  For example, if I only want HD conferencing, I don't want to request that in a hint and be given 640x480 greyscale.
> Our next thought was to give hints but, for each hint, also give a minimum that we would accept.  The problem with this is two-fold:  first, we would need to add additional syntax to differentiate between what we would like (the hint) and what we would accept (the minimum), and second, this doesn't address the either-or case.  An example of the latter would be wanting 1080p video unless it wasn't available, in which case you wanted only a thumbnail video.
> At this point we realized that application authors will *always* want control over what they get.  Since we will be using a hints API that does not guarantee you will get exactly what you request, the next best option is to be able to specify configurations/profiles that would be acceptable, such that the browser is expected to give you any *one* of the configurations you would be happy with.  To increase the chances that the browser will give you something close to what you request and, most importantly, a configuration with options that are internally consistent with respect to your application, without a low-level API the easiest way to achieve this is to have the browser itself tell you in advance (via getCapabilities) exactly which configurations/profiles of hints it can understand and/or provide.
> 
> This led us to propose that 1) the return from getCapabilities() should be a collection of profiles/configurations that the browser is capable of providing, and that 2) the input to the Hints API should be an author-ordered subset of these configurations.  While it does not guarantee in advance which configuration the author will get, it does guarantee that if an acceptable configuration can be provided it will be, without having to try, fail, try again, fail again, etc.
> 
> A side benefit of this approach is that if a browser becomes able to provide some new capability, by returning that in the getCapabilities() call it becomes available to the app author.
> 
> 
> The specific examples we show below are not in any way intended to specify the Hints API, but rather to show what a JSON array of profiles/configurations might look like.
> 
> -- dan
> 
> 
> On Jan 19, 2012, at 11:04 AM, Dan Burnett wrote:
> 
>> Here is a proposal for a Capabilities API.  Note that the syntax of what is returned will ultimately be determined by the Hints API.  I have included a brief example return value and answers to questions I expect people will have.
>> 
>> -- dan
>> 
>> 
>> 
>> interface Capabilities {
>> HintList getCapabilities()
>> };
>> 
>> The getCapabilities method returns a JSON array of profiles representing local capabilities, where each profile is a collection of internally consistent options.  The intent is that the author can select the profiles that are acceptable, rank them in order from most to least preferred, and send the ordered list of acceptable profiles as input to the Hints API.  The Hints API will define the options that are allowed within profiles, as well as defining the HintList type.
>> 
>> Examples:
>> (Full examples are TBD when we have a Hints API.  These are just to give an idea of what the structure would look like)
>> 
>> This is an example of a HintList, a list of profiles which the app author will subset and priority-order before sending to the Hints API.
>> 
>> [{Video:
>>   {Purpose: whiteboard,
>>    Resolution: high},
>> Audio:
>>   {Type: speech,
>>    Resolution: mobile,
>>    Handsfree: true}},
>> {Video:
>>   {Purpose: action,
>>    Bandwidth: lan-internet},
>> Audio:
>>   {Type: music,
>>    Bandwidth: broadband-high,
>>    Handsfree: false}},
>> {Video:
>>   {Purpose: face2face,
>>    Resolution: 720p},
>> Audio:
>>   {Type: speech,
>>    Bandwidth: broadband-medium,
>>    Handsfree: true}},
>> {Video:
>>   {Resolution: 1080p,
>>    Bandwidth: broadband-low},
>> Audio:
>>   {Resolution: landline,
>>    Bandwidth: streaming-high,
>>    Handsfree: false}},
>> {Video:
>>   {Purpose: action,
>>    Resolution: 480p},
>> Audio:
>>   {Resolution: fm,
>>    Bandwidth: streaming-low}},
>> {Video
>>   {Purpose: face2face,
>>    Resolution: medium},
>> Audio:
>>   {Type: music,
>>    Resolution: hi-fi}},
>> {Video
>>   {Purpose: whiteboard,
>>    Resolution: low},
>> Audio:
>>   {Type: speech,
>>    Resolution: studio}},
>> {Video
>>   {Purpose: face2face,
>>    Resolution: high}},
>> {Audio:
>>   {Type: music,
>>    Bandwidth: broadband-high}}]
>> 
>> 
>> FAQ
>> Why do we need a Capabilities API?
>> 1. Web developers want to be able to grey out/hide capabilities that are not available, such as a webcam or microphone.
>> 2. Web developers may not know which configurations are allowed or feasible.  For example, for a video device it may be possible to specify high quality or low bandwidth, but not both.  It is quite likely that an app author who wants both would rather specify a preference for one than trust the hint system to make the right selection between the two.
>> 
>> Why use a list of profiles to select from, order, and send to the Hints API?
>> 1. This ensures that the profiles provided as hints are feasible and contain compatible and legal options.
>> 2. It allows the author to see precisely the levels of detail available as hints.  For example, one video device might allow resolutions to be specified as 'high', 'medium', 'low', '640x480', and '1024x768'.  Even though these options are not all mutually exclusive, they are all valuable since some authors may wish to use the descriptive labels and others the precise resolutions.
>> 3. This may allow for easy extensibility.  For example, if a video device supports 'avatar mode' and a browser supports that capability, it will appear in the capability profiles and be selectable via hints on that browser.
>> 
>> What about remote capabilities?
>> 1. Remote capabilities can be obtained through separate negotiation between the parties, since each end can obtain its own local capabilities and then communicate, e.g., via the data channel.  There is no need to standardize a specific protocol for obtaining remote capabilities.
>> 2. By leaving the communication of capabilities up to the Javascript application, the application can control how much information is made available to the remote end, thus implementing privacy restrictions appropriate to the application.  The browser can always, of course, restrict the information it provides to ensure that even the local side does not have inappropriate capability information.
> 
>