- From: Chris Wilson <cwilso@google.com>
- Date: Mon, 18 May 2015 12:47:25 -0700
- To: Joe Berkovitz <joe@noteflight.com>
- Cc: "Hofmann, Bill" <bill.hofmann@dolby.com>, Audio Working Group <public-audio@w3.org>
- Message-ID: <CAJK2wqUJAFG8XC81vTpDnRC53STM8sXYxZn1a5=PLKWSARUt1Q@mail.gmail.com>
SGTM. On Mon, May 18, 2015 at 10:52 AM, Joe Berkovitz <joe@noteflight.com> wrote: > Thanks Bill and Chris for the additional thoughts. I also think we need a > little more clarity about how enumerating devices with enumerateDevices() > is supposed to work, too. > > Look at the fine print (step #4) here: > > http://w3c.github.io/mediacapture-main/#dom-mediadevices-enumeratedevices > > It appears that if the user doesn't grant permission to *at least one > device* in the result returned by enumerateDevices() (which doesn't take a > query constraint argument) , then the returned list doesn't include any > names for the devices -- they get censored by a filtering step. This is > presumably an anti-fingerprinting measure: there needs to be some UA/user > interaction before a site's scripts can get access to that list of devices. > > If this behavior is taken as given -- and I think it may be hard to argue > otherwise -- then it appears the only workable approach is this: > > 1. Call getUserMedia() with some constraints, to try to get permission to > at least one default output device. (Presumably the app supplies a set of > constraints that were tuned to choose a reasonable default.) > > 2. If permission is granted by the user, call enumerateDevices() to obtain > a full, user-readable list of device names for output devices. > > 3. In whatever device-choice UI is offered by the app, display the full > list of device names from step #2 to the user, defaulted to the device > chosen by step #1. > > I guess the good news here is that the constraints to getUserMedia() are > relatively powerful. Sample rate is already a constraint attribute and > latency is under consideration. If MCTF will include channel count (which > seems pretty uncontroversial, and fingerprinting is not really a > consideration here), perhaps that suffices to get us off the ground. > > I don't think it's too awful for the enumerated device list to be > relatively unconstrained in nature. As long as the default one is a > reasonable choice the user can do an OK job of picking an alternative given > the set of valid choices. > > So my proposal is to get back to the group and suggest the inclusion of > channel count as a constraint, if I don't hear objections. > > Thoughts? > > > ...Joe > > > > On Mon, May 18, 2015 at 12:14 PM, Chris Wilson <cwilso@google.com> wrote: > >> I think number of channels and sample rate are the most critical. Next >> up would be latency and "binaural delivery" - aka "headphones" - as that >> can indicate that HRTF, etc are appropriate (although I'd point out that >> attribute can change without affecting the rest of the device, so maybe >> it's a separate mechanism?). I think HDMI is a red herring. >> >> On Sat, May 16, 2015 at 12:07 PM, Hofmann, Bill <bill.hofmann@dolby.com> >> wrote: >> >>> Joe: >>> >>> >>> >>> Thanks for your notes on this. When I think about use cases: >>> >>> >>> >>> 1. A user wants to connect their device (e.g., a digital media >>> adapter) to an AV Receiver so they can play a game and take advantage of >>> their surround system. DMAs are starting to also be game consoles now, many >>> in China and most recently NVIDIA’s new device. No reason why they >>> shouldn’t support HTML games, and HTML is often the UI for these devices >>> >>> 2. A user wants to play a game with a headset – knowing that the >>> device is connected to a headset jack at least would allow a game to do a >>> headphone render >>> >>> 3. A user wants to watch a movie, and the HTML player wants to >>> adapt the audio properly based on the rendering device >>> >>> >>> >>> It’s most likely, to me at least, that the user would chose the device >>> to render to, **though**, you’d really want the default choice to be >>> the “best one”. So that does suggest that at the very least, you should be >>> able to: >>> >>> · Determine the number of outputs (if == 1, the choice is easy J >>> ) >>> >>> · Identify the type of output (speaker, headphone, HDMI) >>> >>> · The number of channels >>> >>> without permission. >>> >>> >>> >>> Then, the first time (or if the configuration changes), the user would >>> be asked for permission to use the output device, and potentially be given >>> a list of choices beforehand based on the info above, which ought to be >>> enough. It’s probably fine to get the rest of the characteristics later. >>> I don’t recall where getUserMedia ended up with respect to permissions – >>> it’d be deadly to have to configure each time you turn on your DMA or >>> launch a different app, but that doesn’t relate to this problem. >>> >>> >>> >>> I think the constraints approach is fine, but realize that people will >>> use that as a way of enumerating – if you ask for stereo-capable outputs, >>> for instance. I don’t think you can count on always only getting one >>> output. And agree on Chris’ concern. The way you’d probably end up having >>> to code this if you wanted headphones but could deal with speakers (for >>> instance) would end up being a set of getUserMedia calls with constraints, >>> and taking the first. Unless the constraint could be an OR. I foresee a >>> need for guidance about the right way to code this sort of thing. >>> >>> >>> >>> -Bill >>> >>> >>> >>> *From:* Joe Berkovitz [mailto:joe@noteflight.com <joe@noteflight.com>] >>> *Sent:* Friday, May 15, 2015 8:25 AM >>> *To:* Audio Working Group >>> *Subject:* Re: Web Audio WG feedback LC-3023 (Re: Media Capture and >>> Streams Last Call review) >>> >>> >>> >>> Before responding to Harald, I'd like to solicit some discussion within >>> the Audio WG. I think the most important questions here are: >>> >>> >>> >>> 1. If we want to be able to find out properties of devices in an >>> enumerated list without requesting device access from the user, then what >>> is the absolute "must have" set of properties for Web Audio to include in >>> enumerateDevices() results? The more we ask for, the less likely we will >>> get them -- and some may be more likely to generate long debates than >>> others, like HDMI. >>> >>> >>> >>> 2. Do we need to enumerate devices, or is it OK for us to use >>> getUserMedia() with constraints on these properties, and then pass the >>> deviceID of the returned mediaStream -- obtained with >>> mediastream.getCapabilities() -- that matches those constraints to an >>> AudioContext constructor? (As opposed to using >>> createMediaStreamDestination(mediaStream) which would have the various >>> sample rate issues raised by Chris). >>> >>> >>> >>> >>> >>> ---------- Forwarded message ---------- >>> From: *Harald Alvestrand* <harald@alvestrand.no> >>> Date: Fri, May 15, 2015 at 7:12 AM >>> Subject: Web Audio WG feedback LC-3023 (Re: Media Capture and Streams >>> Last Call review) >>> To: Joe Berkovitz <joe@noteflight.com>, Stefan Håkansson LK < >>> stefan.lk.hakansson@ericsson.com>, public-media-capture@w3.org, Audio >>> Working Group <public-audio@w3.org> >>> >>> >>> Hello, and thanks for your input! >>> >>> I'm seriously in two minds about this - on one hand, it seems like >>> functionality that is well worth having. >>> >>> On the other hand, it seems like a long list of things that could be of >>> interest here, and I can easily envision considerable time passing while >>> we discuss the details of each (for instance, if we expose the fact that >>> an output is HDMI, we also expose the fact that it's either crypto >>> capable or not crypto capable....) >>> >>> I think a lot of things can be addressed within the >>> capabilities/constraints/settings model we've adopted for getUserMedia - >>> one can define new constraints that get you the selectivity you want, >>> one can call getCapabilities() to figure out what kind of device one >>> has, one can use getSettings() to figure out what the current state of >>> play is. If so (and if the TF keeps the "registry" approach for >>> constraints), solving these problems can be as easy as authoring an >>> add-on document called "additional audio capabilities and constraints". >>> >>> But I'm not sure if that will cover all your needs, or if this is the >>> most elegant way of doing it - certainly some will make immediate note >>> that the constraints mechanism isn't what they consider elegant. >>> >>> What do you see as the best way forward here - aim to address this >>> later, or do we have parts of this problem that we *have* to address now? >>> >>> Harald >>> >>> >>> Den 21. april 2015 20:59, skrev Joe Berkovitz: >>> > Hello Stefan, >>> > >>> > >>> > Thank you for your recent solicitation of feedback to on the Media >>> > Capture and Streams API, which I passed to the Web Audio Working Group. >>> > >>> > >>> > The Web Audio WG so far has identified one key item that we would like >>> > to see addressed. The MediaDeviceInfo result from enumerateDevices() >>> > ( >>> http://www.w3.org/TR/2015/WD-mediacapture-streams-20150414/#idl-def-MediaDeviceInfo >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_TR_2015_WD-2Dmediacapture-2Dstreams-2D20150414_-23idl-2Ddef-2DMediaDeviceInfo&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=TljDhqSLOq4OTqBOCkRFi2iTNGDcdg0vbZ9A-vrOnlw&e=> >>> ) >>> > lacks information that is typically available in the underlying OS >>> > implementations that we think would be very helpful for >>> implementations: >>> > >>> > __ __ >>> > >>> > __· __Channel count and configuration (Mono, Stereo, 5.1, 7.1, >>> > etc…)____ >>> > >>> > __· __Physical Output (Headphone, Speaker, HDMI, …)____ >>> > >>> > __· __Latency (this matters a lot for gaming -- it will be very >>> > low for on-board hardware, perhaps quite high for wireless audio >>> > bridging like Apple TV)____ >>> > >>> > __· __Output capabilities (bitstream passthrough vs PCM – >>> > relevant in digital media adapter cases (Chromecast, etc))____ >>> > >>> > >>> > It is perhaps sufficient from a user interface point of view to have a >>> > string to display, but for a program to be able to either adapt to the >>> > user selection or to guide and default the user selection, the above >>> are >>> > pretty important characteristics, at least in some use cases. Many if >>> > not most of the host OSes that user agents run on expose these sorts of >>> > output device characteristics. ____ >>> > >>> > >>> > Aside from the difficulty with enumerating devices, there is also >>> > perhaps a need to make it possible for applications to query the set of >>> > available devices with respect to the above >>> > charateristics. MediaTrackConstraints and MediaTrackSettings do not >>> > currently include constraint attributes that map to items in the above >>> > list. And even if they do, arriving at a practical goodness-of-fit >>> > metric that can be generalized across a spectrum of audio apps may be >>> > difficult. >>> > >>> > >>> > >>> > The same concerns apply to the set of input devices.__ >>> > >>> > __ __ >>> > >>> > Please let us know if this issue makes sense to the group and can be >>> > addressed within the timeframe of the coming run-up to a Last Call WD. >>> > We'd be happy to arrange some sort of inter-WG call to try to make >>> > progress on this together. >>> > >>> > >>> > Thank you! >>> > >>> > >>> > Best regards, >>> > >>> > >>> > Joe Berkovitz >>> > >>> > co-chair Web Audio WG >>> > >>> > >>> > *Noteflight LLC* >>> > Boston, Mass. >>> > phone: +1 978 314 6271 >>> > www.noteflight.com >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=> >>> <http://www.noteflight.com/ >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com_&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=TX3MVside5EU5bm_UNZyg2r1SdoBFsm-f8nP7K1k4Y8&e=> >>> > >>> > "Your music, everywhere" >>> > >>> > On Wed, Apr 15, 2015 at 1:31 AM, Stefan Håkansson LK >>> > <stefan.lk.hakansson@ericsson.com >>> > <mailto:stefan.lk.hakansson@ericsson.com>> wrote: >>> > >>> > The WebRTC and Device APIs Working Groups request feedback on the >>> Last >>> > Call Working Draft of Media Capture and Streams, a JavaScript API >>> that >>> > enables access to cameras and microphones from Web browsers as >>> well as >>> > control of the use of the data generated (e.g. rendering what a >>> camera >>> > captures in a html video element): >>> > http://www.w3.org/TR/2015/WD-mediacapture-streams-20150414/ >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_TR_2015_WD-2Dmediacapture-2Dstreams-2D20150414_&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=iqvKvUbbXBvyilFPRoiU-moSntiBDqoGqKdbqREA2EY&e=> >>> > >>> > The groups have identified the following other W3C Working Groups >>> as >>> > likely sources of feedback: >>> > >>> > - HTML Working Group, especially the HTML Media Task Force, as our >>> API >>> > extends the HTMLMediaElement interface and defines a new type of >>> media >>> > input via MediaStream >>> > >>> > - WebApps Working Group, especially on the overall usage of Web >>> IDL and >>> > the definition of error handling >>> > Audio Working Group, as the Web Audio API builds upon the >>> MediaStream >>> > interface >>> > >>> > - WAI Protocol and Formats Working Group, especially on the impact >>> of >>> > the user consent dialog and the applicability of the indicators of >>> > device usage in assistive tools >>> > >>> > - Web and TV Interest Group, as the manipulation of media input >>> can be >>> > relevant to some of their use cases (e.g. glass to glass) >>> > >>> > - Web App Security Working Group, especially on our links between >>> > secured origins and persistent permissions, and our current policy >>> with >>> > regard to handling access to this "powerful feature" >>> > >>> > - Web Security Interest Group, especially on our security >>> considerations >>> > Privacy Interest Group, as access to camera and microphone has >>> strong >>> > privacy implications >>> > >>> > - Technical Architecture Group, for an overall review of the API, >>> > especially the introduction of the concept of a IANA registry-based >>> > constraints system, the use of promises, and our handling of >>> persistent >>> > permissions >>> > >>> > We naturally also welcome feedback from any other reviewers. >>> > >>> > The end of last call review for this specification is set to May 15 >>> > 2015; should that deadline prove difficult to meet, please get in >>> touch >>> > so that we can determine a new deadline for your group. >>> > >>> > As indicated in the document, comments should be sent to the >>> > public-media-capture@w3.org <mailto:public-media-capture@w3.org> >>> > mailing list. >>> > >>> > Thanks, >>> > >>> > Frederick Hirsch, Device APIs Working Group Chair, >>> > Harald Alvestrand and Stefan Hakansson, WebRTC Working Group >>> Chairs and >>> > Media Capture Task Force Chairs >>> > >>> > >>> > >>> > >>> > -- >>> > . . . . . ...Joe >>> > >>> > *Joe Berkovitz* >>> > President >>> > >>> > *Noteflight LLC* >>> > Boston, Mass. >>> > phone: +1 978 314 6271 >>> > www.noteflight.com >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=> >>> <http://www.noteflight.com >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=> >>> > >>> > "Your music, everywhere" >>> >>> >>> >>> >>> >>> -- >>> >>> . . . . . ...Joe >>> >>> >>> >>> *Joe Berkovitz* >>> >>> President >>> >>> >>> >>> *Noteflight LLC* >>> >>> 49R Day Street / Somerville, MA 02144 / USA >>> >>> phone: +1 978 314 6271 >>> >>> www.noteflight.com >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=> >>> >>> "Your music, everywhere" >>> >> >> > > > -- > . . . . . ...Joe > > *Joe Berkovitz* > President > > *Noteflight LLC* > 49R Day Street / Somerville, MA 02144 / USA > phone: +1 978 314 6271 > www.noteflight.com > "Your music, everywhere" >
Received on Monday, 18 May 2015 19:47:54 UTC