Media Capture Task Force Teleconference

09 May 2012


See also: IRC log


Bryan_Sullivan, Cathy, Dan_Burnett, Jim_Barnett, Josh_Soref, Stefan_Hakansson, adambe, anant, derf, dom, fjh, hta, jesup
Stefan_Hakansson, Harald_Alvestrand


<trackbot> Date: 09 May 2012

<anant> Zakim: Mozilla is anant

<scribe> Scribe: Josh_Soref

Scribe Selection

stefanh: Josh_Soref, will you scribe?

Josh_Soref: yes

Agenda Adjustments

stefanh: I'd like to add a slot for burn to talk about Constraints

Minutes Approval

burn: I sent a minor correction to Josh_Soref just now

Josh_Soref: I'll update that and send out new minutes

stefanh: is it ok to approve the minutes with that correction?

RESOLUTION: Minutes from 24 April are approved with burn's correction

stefanh: there were 2 actions from the minutes
... burn to write a proposal
... and myself to contact travis
... i haven't done that yet

Audio/Video Mandatory Syntax Proposal

burn: i sent an email, about an hour ago
... to the list
... I promised to provide an email with some of the syntax alternatives
... these two, I think represent as much of the consensus
... and interest as I heard
... there were minor variations on these
... I think overall, if we could choose one of the three approaches
... the original, alternative 1, and alternative 2
... but it'll be clear what we're discussing
... the original is copied directly out of the constraints proposal

<dom> Syntax options for constraints structure Dan Burnett

[ Alternative 1 ]

burn: here, i restructured it
... instead of mandatory and optional being the top-level keys
... the top level are video and audio
... with mandatory/optional underneath them
... or for the value of the video key to be "true"
... I had to come up with a value of "optional"
... video is a struture
... with mandatory and optional parameters
... mandatory is an object (dictionary)
... and optional is still a list (array)
... ordered
... the ones that come first have a higher priority

[ Example 2 ]

burn: this uses the syntax shortcut for optional

[ Example 3 ]

burn: this uses the syntax shortcut for mandatory

<Zakim> Josh_Soref, you wanted to note that "optional" would need to be a string

anant: Josh_Soref noted that optional would need to be a string
... we could make them be boolean or dictionary
... and for optional you could not specify the property

burn: that wouldn't be the same
... {audio:optional} means, I'm ok with audio, but i don't need it
... if you say {}, audio means you don't want audio

anant: we'll have to think about this a bit more
... from an implementation perspective
... they could be Boolean, String, or Dictionary
... we might just have String or Dictionary
... that's just a minor detail
... overall, it looks reasonable

adambe: perhaps, instead of making all three stirngs
... could we use an enumeration type?

burn: we could use {video:{mandatatory:{}}}
... Dict + Enum

<Zakim> Josh_Soref, you wanted to note that "enum" is being deprecated

anant: we're trying to move away from enums
... enums become messy things
... navigator.media.ALLOW...
... so strings are better
... we're moving away from them for PeerConnection

burn: any other questions about alternative 1?

[ None ]

[ Alternative 2 ]

burn: same 3 examples
... video and audio have been added to the dictionary

<anant> Josh_Soref: enums are too verbose and require a prefix everytime (like navigator.media) that adds unnecessary characters

burn: we could still do...
... the key difference is that we have {mandatory:{}, optional:{}} at the top
... In the second example of alternative 2, we could modify optional:
... to be an optional value for audio/video as in alternative 1

<dom> (enum is actually a construct in WebIDL to build list of acceptable strings)

adambe: a key difference here
... is that we don't have to prefix the constraints w/ video/audio in Alternative 1
... but they're necessary in Alternative 2
... in A2E1
... is video mandatory in this example, because there's a mandatory Video named constraint
... in Example 1

burn: it isn't needed
... if you look at the algorithm
... it walks through to determine if video is requested
... The fact that any video constraint exists under mandatory means that video is required
... if it was only requested in optional, then it was optional

XZ: wouldn't there be a way to say
... "if i get video, i need video with this properties"

burn: i'm not sure if there's a way to optionally request audio with constraints

anant: is there a UC for this?

burn: I can't think of one
... i can't think of a case where you'd say "i'd like audio", and have it optional

anant: we need a bit more thought over there

burn: i could live with that
... i can think of optional constraints
... but not on getting/not getting the stream

<bryan> is the optionality in at least some use cases intended to allow UI to enable the user to decide?

anant: between example 1 and 2, it seems inconsistent
... are the top level keys {mandatory:{}, optional:{}, video:{}, audio:{}}

burn: in Alternative 2, but not Alternative 1

bryan: it seems one UC for optionality is to let the User choose
... the app could work just as well w/ no audio
... you don't know if the user is a Signing user or a Speaking user

anant: you're right
... you could start with Video for Video chat
... and add audio later

burn: this is interesting
... because none of these syntaxes support audio optional, but with constraints

<anant> or, more likely, start audio and "upgrade" to video :)

burn: or video optional but with constraints
... the fact that you have the object
... is essentially saying... you expect to get the stream

<bryan> I'm not sure about the constraints importance in an optional context

burn: however, if they want an optional audio stream
... the reason it's optional is so the user can say "no"
... that doesn't change the desire of the application to request parameters for a stream it receives

Josh_Soref: +1

<bryan> ok, understand better now... +1

adambe: perhaps you could have

burn: so Audio + Video keys
... indicate mandatory / optionality of the stream itself

adambe: then you can have the same constraint set

burn: one disadvantage is that you go back to the longer constraint names
... for a moment, i thought we were about to agree on the approach of alternative one

adambe: sorry for that

burn: i never intended to provide an opportunity for a stream to be optional
... only for constraints to be optional
... i'll probably have to go back and think about this some more
... in order to have it work in the ways we've discussed
... to support optional streams
... i'm not saying it couldn't be done
... but it's non-trivial to address
... so, i'm asking: how important is that?
... i don't think we've ever discussed optional streams

anant: because before we had hints
... and everything was optional
... i think you're fine for deferring this for now
... and maybe we discuss it further on the ML

stefanh: the chairs sent a message last week
... saying we thought it was time to
... BBB
... but maybe we need more thinking time

adambe: we could put it in the draft
... and if we pick alternative 1 or 2
... and we could discuss optional streams later

stefanh: in this meeting, is alternative 1 or 2 preferred

<anant> I prefer alternative 1

burn: without optional streams, i'd prefer alternative 1
... with optional streams, i think alternative 2

anant: i think we could put alternative 1 in now
... and when we rethink things, we could change it

adambe: i think optional streams is written in the notes
... shouldn't it be optional TRACKs?

burn: yes
... you're correct
... i wrote this when we weren't clear
... what will go into the document will very quickly need to change to Track
... if i say Stream, i mean Track
... i guess we could do a quick check
... anant, i see you prefer alternative 1

anant: yep

burn: i'm fine iwth that

Josh_Soref: I think I favor alternative 1

burn: I also favor alternative 1

hta: i think we have consensus to put alternative 1 in the draft
... i will adjust the enums to strings

PROPOSED ACTION burn to write alternative 1 into the draft

stefanh: dom, Actions will go into another tracker?
... for media capture?

dom: yes

<hta> ACTION: burn to write alternative 1 into the draft [recorded in http://www.w3.org/2012/05/09-mediacap-minutes.html#action01]

<trackbot> Created ACTION-1 - Write alternative 1 into the draft [on Daniel Burnett - due 2012-05-16].

Resource Reservation

stefanh: i sent an email about this
... about whether it would reserve
... initially or when it's used
... there was some discussion on the list
... there's some argument for an application to ask at load time for permission
... but there's a drawback
... "what does a media stream with an enabled track mean?"
... another web/native application could use it

hta: that's not a very clear item
... we don't know if a camera/microphone has exclusive access
... in Chrome today, any number can have the camera open
... so far we haven't implemented constraints
... my assumption, is ideal
... we'd take the

derf: there's of course the possibility that an app other than chrome takes control of the camera
... between request, and attaching to a destination

anant: one thing we're experimenting with
... is having getUserMedia reserve the device
... but not activate it
... if we can't reserve the device, the call fails
... if video was mandatory
... we don't activate the device
... the camera isn't turned on
... and no data is sent
... but when the stream is tied to a canvas/MediaStream
... it is turned on
... we haven't tackled the question of allowing more than one application to have access to the camera

<bryan> Android does not support multiple app access to camera AFAIK

anant: it might have unintended user consequences

<adambe> derf

derf: the thing that concerns me, is that the user could forget the app that asked for permission

anant: the way i planned to address that

<bryan> reservation is an interesting feature but needs to be clear to the user in the permissions UI

anant: is that we'd tell the user that a second thing is asking
... indicating which is using it currently

derf: i'm less concerned with that case
... than with the case of a single page asking
... and not using it for a while
... and the user forgetting

<bryan> the error callback would need to clarify that a reservation error occured

derf: and then the app later turning it on

hta: we're thinking of tray notification

anant: we could show the video active in the tab notice
... and disband when the tab close

adambe: i don't care about copying video buffers around

hta: the distinction is that when you call gUM
... the tray icon appears
... and when you connect to a sink, the light turns on

jesup: not every camera has a light to turn on

hta: right, and some of them are under software control

bryan: if we give the requesting app some indication
... that it failed because it was reserved
... that would let it give some UI to the user
... and the UI for the user on a reservation would need to be clear

anant: i agree
... especially for incoming calls
... if you're on a page, and aren't currently using the camera
... but if you get a call,
... you may want to accept the call and drop the camera
... if the Browser can't get access
... then it'll fail
... but if the Browser has two apps
... then it can switch internally

bryan: if i'm using a code-scanning
... and i get a call
... i wouldn't want the app to fail

anant: i'm describing getting a case which lets the user
... transfer the video from one browser app to another

adambe: if we can't tell the User that the app is currently using it
... in Windows, you always get this frustrating thing
... "the file is in use"
... we have a request, a reservation, and a start reading phase
... talking about reservation
... you don't get the benefits
... the stream can't monitor the underlying source
... if you only reserve
... you have drawbacks
... but you can't monitor the stream

anant: i don't fully understand that

<bryan> oops

<bryan> or me

anant: for practical purposes,
... get ... reservation
... is made
... it's an optimization
... with no sink, we don't need to move frames around
... you can observe it
... but we haven't turned on the camera

adambe: throwing away frames isn't a good thing
... on mobile, that's bad for your battery
... for a long running application
... (chat)
... you want to be able to make a reservation

anant: for a gUM
... the reservation is an internal detail
... for all practical purposes
... if gUM succeeds, you will be able to get frames
... because you have a hold

adambe: it's a bit different than a native app

anant: the incoming call case is different from the chat case
... we want to discourage pages from eagerly requesting

stefanh: i agree with anant
... this seems like the cleaner approach

<Zakim> Josh_Soref, you wanted to note that modern Windows says which app and has a "switch to"

anant: maybe we can write this up and send it to the list
... and if we get consensus, we can draft it into a document
... this may be an authoring guide
... if someone calls eagerly
... and another app asks
... the UA can let the User terminate the early request

adambe: talking about the Stream API
... this would make it more consistent with a network stream
... you have a natural flow all the time
... someone is sending data
... it would make local+remote streams more similar

anant: that seems like a good side-effect

Several Active Devices

stefanh: we touched on this briefly at the last call

anant: what do we mean by several devices
... is that several mic's + cameras?

stefanh: if you have 2 cameras and want to use both
... do you call gUM multiple times
... or do you get multiple Tracks?

anant: I think we need more UCs in the Scenarios doc
... and even if we have this UC
... how important is this UC?
... we could provide additional Constraint
... changing gUM depends on how important we think this UC is
... if we don't change this, we let the App call gUM multiple times
... where they get 2 MediaStreams, one camera per stream
... in the first case, you get 1 MediaStream with multiple tracks
... it depends on how important the UC is
... and we could just use Constraints

stefanh: you could combine multiple MediaStreams into one

anant: yes, using the MediaStream Constructor
... it depends on input from developers

jesup: how do you determine cameras?

anant: it's user choice

burn: we haven't talked about distinguishing

anant: what is the type of distinction?

burn: user/env front/back

anant: you could have "rotatable cameras"
... that's fuzzy

burn: i agree
... and that's independent of how you specify it

anant: I'm leaning to have multiple gUM calls
... one per camera
... it depends on how important it is
... how frequent we expect this to be

adambe: i think the specification doesn't say anything about this
... and the fact that MediaStreams can contain several streams of each type
... I think we need to specify that the UI should only allow the user to select one of each type

<Zakim> Josh_Soref, you wanted to note that you could have 3 identical cameras that are only distinguishable by the user

burn: at some point we need to make a decision
... if we decide that gUM can only return one Track of each type
... then the way you're describing it is how we should be explicit in the spec

adambe: LocalStreams could only have one of each type
... an empty list is something
... a list of one thing is something

stefanh: i don't know if we should take this to the list
... at this point i'm leaning toward
... I guess we should take this to the list again

Stored Permissions

stefanh: hta, you proposed this

hta: "this application can use my camera any time"
... if you have stored permissions
... you need a UI for managing them
... Chrome for picking which camera to use
... is only invoked at checking permissions
... then if you've given an app permission
... how does the app get back to the Camera selection dialog?

jesup: I think you need to be able to have a way to select the selection dialog regardless of this
... wanting to have a way to flip between front and back on a mobile device
... that problem has to be addressed anyway
... possibly another gUM call
... our thinking on this has been
... having ongoing permissions tied to app installation

hta: yes, that's one way of managing
... if you have an app permission mechanism
... and everything else lives with temporary permissions

jesup: that's our current way of thinking

hta: for getting the dialog when you wouldn't normally need to have them
... is to have a constraint "really ask the user"

burn: constraints for asking permissions
... i don't understand

hta: something in the constraints dictionary that says
... "ask user to pick device"
... ask-user-yes

adambe: app has permissions
... but wants to prompt?

hta: app has permission for front
... but wants the back camera

adambe: i guess the browser database needs to specify which camera the app has been authorized to use
... then if you use gUM
... you can get a prompt for the camera you didn't previously authorize

hta: so a db w/ each camera is its own permission domain?

adambe: it doesn't sound nice
... but perhaps
... i wouldn't be comfortable with the application being able to freely swich
... if i gave permission for one
... for it to switch from one, to the next, to the next

jesup: if you gave the app permission to all of your cameras?

adambe: i guess so, it'd be a different kind of permission

jesup: the only ones you'd be giving this to
... are ones you trust at a desktop/plugin install
... which gives full access
... i don't know if there's something to be gained for such fine grained control

adambe: i agree with you
... it's only risky applications on the web

jesup: for stored applications
... because of the trust you're giving them
... i think All or None is enough
... it simplifies the UI

adambe: that kind of application wouldn't use this UI at all
... it could put up a self view of all cameras
... and say "which do you want to use"
... in gUM, it's selection + permission

jesup: correct, someone with that kind of app might want to do it themselves

<Zakim> Josh_Soref, you wanted to cry

Josh_Soref: ...
... there are cases where a user wants to use a "Camera" app
... but doesn't realize it's going to randomly snoop on the other camera
... There's also the fact that the App wants to use the system picker
... because it doesn't want to build the UI itself
... or because it wants to have the standard system look and feel for the camera picker

jesup: I agree with the second case
... I don't think there are many env- only camera applications
... that would never want to access the other camera
... e.g. instagram would probably want to be able to use either camera
... there might be a UC for a native UI to toggle cameras

adambe: do you think there might be a UC for
... a toggle button

jesup: I think for a dedicated app, like skype
... an in App button is pretty common

adambe: in the other case
... toggling from an external ui
... wouldn't it be problematic
... if you send a stream with a resolution/framerate
... and you toggle
... and the JS application gets a different frame-rate/resolution
... do you think that should be invisible for the app

jesup: i would have no problem with the application getting an event notification
... i think the system needs to be able to deal with those changes anyway
... and applications need to be able to deal with them anyway
... if need be, changing resolutions on streams
... i don't see a problem there
... there could be ways where it happens now
... if the user supplies a Video file instead of a Camera
... the file may have resolution/framerate changes in it
... you have no ability to change it, you have to adapt to it

adambe: it's quite important to be able to mute the camera from the browser ui

jesup: absolutely, critical
... because you can't trust the app to do it

adambe: that would also be a stream change

jesup: the camera may stop sending video
... change its frame rate
... you can't count on the camera to keep a constant frame-rate

<Zakim> Josh_Soref, you wanted to note that users don't read permissions dialogs - ever

hta: i think we have gotten a good handling of the issues
... i don't see a concrete suggestion about language
... anyone else want to speak on this topic?
... stefanh: any other topics?

stefanh: there is quite a long list of open items
... i think we need to start addressing them
... any input would be valuable

<stefanh> http://www.w3.org/wiki/Media_Capture#Open_Items

hta: the WebRTC meeting is coming up
... June 11 in Stockholm
... followed by 2 days of RTCWeb meetings

<dom> WebRTC Meeting June 11

Josh_Soref: there was some mention of MediaStream in the HTML WG F2F last week, I hope to send minutes for it later this week

<hta> ACTION: anant to describe that getUserMedia reserves the devices, causing the "camera to appear busy" [recorded in http://www.w3.org/2012/05/09-mediacap-minutes.html#action02]

<trackbot> Created ACTION-2 - Describe that getUserMedia reserves the devices, causing the "camera to appear busy" [on Anant Narayanan - due 2012-05-16].

<hta> ACTION: anant to add a UI guidelines about "camera is busy because of X" when another request is made [recorded in http://www.w3.org/2012/05/09-mediacap-minutes.html#action03]

<trackbot> Created ACTION-3 - Add a UI guidelines about "camera is busy because of X" when another request is made [on Anant Narayanan - due 2012-05-16].

Josh_Soref: if app 1 calls getUserMedia
... and gets a camera
... and app 2 calls getUserMedia
... then the user can be shown a dialog with both cameras
... the in use one could appear in use but with a preview
... if the user selects that in use camera
... then the UA can indicate the other Web App that is using it
... allowing the user to steal it

anant: For Desktop cases,
... where the Camera is in use by another app running as the same user on the system
... we could steal it
... but we couldn't if it runs as another user (or more privileged user)

stefanh: I'd like to thanks Josh_Soref for scribing every meeting

<anant> Josh_Soref++

stefanh: really grateful

[ Adjourned ]

trackbot, end meeting

Summary of Action Items

[NEW] ACTION: anant to add a UI guidelines about "camera is busy because of X" when another request is made [recorded in http://www.w3.org/2012/05/09-mediacap-minutes.html#action03]
[NEW] ACTION: anant to describe that getUserMedia reserves the devices, causing the "camera to appear busy" [recorded in http://www.w3.org/2012/05/09-mediacap-minutes.html#action02]
[NEW] ACTION: burn to write alternative 1 into the draft [recorded in http://www.w3.org/2012/05/09-mediacap-minutes.html#action01]
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.135 (CVS log)
$Date: 2009-03-02 03:52:20 $