Re: Web Audio WG feedback LC-3023 (Re: Media Capture and Streams Last Call review) from Hongchan Choi on 2015-05-26 (public-audio@w3.org from April to June 2015)

From: Hongchan Choi <hongchan@google.com>
Date: Tue, 26 May 2015 17:52:27 +0000
To: "Hofmann, Bill" <bill.hofmann@dolby.com>, Audio Working Group <public-audio@w3.org>
Message-ID: <CAGJqXNsc50P7Ak3r63aU3sV0eSuSNbR-JH4gPj3r0SB8oBo_EQ@mail.gmail.com>
I believe this is a must-have feature for us. Last time I tried to hook up
gUM with the multichannel audio interface inside Chromium, I had a hard
time to find out relevant information in the underlying code. Now I know
that is simply because the implementation can ignore the channel count. It
would be super helpful if the channel count is in the spec - because
changing underlying (i.e. hardware layer) code to support such feature is
sort of hard to justify if it isn't clearly specified.

-Hongchan

On Tue, May 26, 2015 at 10:36 AM Hofmann, Bill <bill.hofmann@dolby.com>
wrote:

>  It seems that the latest getUserMedia() draft considers only sample size
> and sample rate for audio – do we expect that channel count will make it in?
>
>
>
> -Bill
>
>
>
> *From:* Joe Berkovitz [mailto:joe@noteflight.com]
> *Sent:* Monday, May 18, 2015 2:03 PM
> *To:* Chris Wilson
> *Cc:* Hofmann, Bill; Audio Working Group
>
>
> *Subject:* Re: Web Audio WG feedback LC-3023 (Re: Media Capture and
> Streams Last Call review)
>
>
>
> It seems I was correct on that point -- see Harald's reply.
>
>
>
> On Mon, May 18, 2015 at 4:36 PM, Joe Berkovitz <joe@noteflight.com> wrote:
>
> As I was saying, it looks from the spec like one must call getUserMedia()
> on some local device in order to get the permission issued, to the point
> where enumerateDevices() returns useful info. Hence the scenario I outlined
> in my earlier email of calling getUserMedia() first.
>
>
>
> I'll ask MCTF for clarification on this point.
>
>
>
> On Mon, May 18, 2015 at 4:30 PM, Chris Wilson <cwilso@google.com> wrote:
>
> I'm afraid I'm a bit lost as to how one might prompt for that permission.
> Ideally we'd be able to trigger sufficient permission elevation in an
> obvious way to be able to, say, look for a 5.1 device.
>
>
>
> On Mon, May 18, 2015 at 1:26 PM, Joe Berkovitz <joe@noteflight.com> wrote:
>
> Chris: It is probably much easier for MCTF to agree to provide this
> information from enumerateDevices() if (just like the readable device
> labels) channel counts etc. are filtered unless there's permission to
> access at least one device in the list.  The list is not really useful
> anyway until the device labels are provided, so this seems reasonable to me.
>
>
>
> We can ask for it anyway, I'm just proposing a result that we can live
> with.
>
>
>
> On Mon, May 18, 2015 at 3:47 PM, Chris Wilson <cwilso@google.com> wrote:
>
> Note it's not just "as a constraint", per se - I want to make sure we can
> enumerate the channel counts (etc.)
>
>
>
> On Mon, May 18, 2015 at 12:47 PM, Chris Wilson <cwilso@google.com> wrote:
>
> SGTM.
>
>
>
> On Mon, May 18, 2015 at 10:52 AM, Joe Berkovitz <joe@noteflight.com>
> wrote:
>
> Thanks Bill and Chris for the additional thoughts.  I also think we need a
> little more clarity about how enumerating devices with enumerateDevices()
> is supposed to work, too.
>
>
>
> Look at the fine print (step #4) here:
>
>
>
> http://w3c.github.io/mediacapture-main/#dom-mediadevices-enumeratedevices
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__w3c.github.io_mediacapture-2Dmain_-23dom-2Dmediadevices-2Denumeratedevices&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=A7v4sR8KByGJfQi7evt4CqIXowTuRgp3-n3YUTJ23eg&s=XxqO_XdSBT2aE5Pc8WkvkqFWbWRgzcAz7iET5dQEsnY&e=>
>
>
>
> It appears that if the user doesn't grant permission to *at least one
> device* in the result returned by enumerateDevices() (which doesn't take a
> query constraint argument) , then the returned list doesn't include any
> names for the devices -- they get censored by a filtering step. This is
> presumably an anti-fingerprinting measure: there needs to be some UA/user
> interaction before a site's scripts can get access to that list of devices.
>
>
>
> If this behavior is taken as given -- and I think it may be hard to argue
> otherwise -- then it appears the only workable approach is this:
>
>
>
> 1. Call getUserMedia() with some constraints, to try to get permission to
> at least one default output device. (Presumably the app supplies a set of
> constraints that were tuned to choose a reasonable default.)
>
>
>
> 2. If permission is granted by the user, call enumerateDevices() to obtain
> a full, user-readable list of device names for output devices.
>
>
>
> 3. In whatever device-choice UI is offered by the app, display the full
> list of device names from step #2 to the user, defaulted to the device
> chosen by step #1.
>
>
>
> I guess the good news here is that the constraints to getUserMedia() are
> relatively powerful. Sample rate is already a constraint attribute and
> latency is under consideration. If MCTF will include channel count (which
> seems pretty uncontroversial, and fingerprinting is not really a
> consideration here), perhaps that suffices to get us off the ground.
>
>
>
> I don't think it's too awful for the enumerated device list to be
> relatively unconstrained in nature. As long as the default one is a
> reasonable choice the user can do an OK job of picking an alternative given
> the set of valid choices.
>
>
>
> So my proposal is to get back to the group and suggest the inclusion of
> channel count as a constraint, if I don't hear objections.
>
>
>
> Thoughts?
>
>
>
>
>
> ...Joe
>
>
>
>
>
>
>
> On Mon, May 18, 2015 at 12:14 PM, Chris Wilson <cwilso@google.com> wrote:
>
> I think number of channels and sample rate are the most critical.  Next up
> would be latency and "binaural delivery" - aka "headphones" - as that can
> indicate that HRTF, etc are appropriate (although I'd point out that
> attribute can change without affecting the rest of the device, so maybe
> it's a separate mechanism?).  I think HDMI is a red herring.
>
>
>
> On Sat, May 16, 2015 at 12:07 PM, Hofmann, Bill <bill.hofmann@dolby.com>
> wrote:
>
> Joe:
>
>
>
> Thanks for your notes on this.  When I think about use cases:
>
>
>
> 1.       A user wants to connect their device (e.g., a digital media
> adapter) to an AV Receiver so they can play a game and take advantage of
> their surround system. DMAs are starting to also be game consoles now, many
> in China and most recently NVIDIA’s new device.  No reason why they
> shouldn’t support HTML games, and HTML is often the UI for these devices
>
> 2.       A user wants to play a game with a headset – knowing that the
> device is connected to a headset jack at least would allow a game to do a
> headphone render
>
> 3.       A user wants to watch a movie, and the HTML player wants to
> adapt the audio properly based on the rendering device
>
>
>
> It’s most likely, to me at least, that the user would chose the device to
> render to, **though**, you’d really want the default choice to be the
> “best one”.  So that does suggest that at the very least, you should be
> able to:
>
> ·         Determine the number of outputs (if == 1, the choice is easy J)
>
> ·         Identify the type of output (speaker, headphone, HDMI)
>
> ·         The number of channels
>
> without permission.
>
>
>
> Then, the first time (or if the configuration changes), the user would be
> asked for permission to use the output device, and potentially be given a
> list of choices beforehand based on the info above, which ought to be
> enough.  It’s probably fine to get the rest of the characteristics later.
> I don’t recall where getUserMedia ended up with respect to permissions –
> it’d be deadly to have to configure each time you turn on your DMA or
> launch a different app, but that doesn’t relate to this problem.
>
>
>
> I think the constraints approach is fine, but realize that people will use
> that as a way of enumerating – if you ask for stereo-capable outputs, for
> instance.  I don’t think you can count on always only getting one output.
> And agree on Chris’ concern.  The way you’d probably end up having to code
> this if you wanted headphones but could deal with speakers (for instance)
> would end up being a set of getUserMedia calls with constraints, and taking
> the first.  Unless the constraint could be an OR.  I foresee a need for
> guidance about the right way to code this sort of thing.
>
>
>
> -Bill
>
>
>
> *From:* Joe Berkovitz [mailto:joe@noteflight.com <joe@noteflight.com>]
> *Sent:* Friday, May 15, 2015 8:25 AM
> *To:* Audio Working Group
> *Subject:* Re: Web Audio WG feedback LC-3023 (Re: Media Capture and
> Streams Last Call review)
>
>
>
> Before responding to Harald, I'd like to solicit some discussion within
> the Audio WG. I think the most important questions here are:
>
>
>
> 1. If we want to be able to find out properties of devices in an
> enumerated list without requesting device access from the user, then what
> is the absolute "must have" set of properties for Web Audio to include in
> enumerateDevices() results? The more we ask for, the less likely we will
> get them -- and some may be more likely to generate long debates than
> others, like HDMI.
>
>
>
> 2. Do we need to enumerate devices, or is it OK for us to use
> getUserMedia() with constraints on these properties, and then pass the
> deviceID of the returned mediaStream -- obtained with
> mediastream.getCapabilities() --  that matches those constraints to an
> AudioContext constructor? (As opposed to using
> createMediaStreamDestination(mediaStream) which would have the various
> sample rate issues raised by Chris).
>
>
>
>
>
> ---------- Forwarded message ----------
> From: *Harald Alvestrand* <harald@alvestrand.no>
> Date: Fri, May 15, 2015 at 7:12 AM
> Subject: Web Audio WG feedback LC-3023 (Re: Media Capture and Streams Last
> Call review)
> To: Joe Berkovitz <joe@noteflight.com>, Stefan Håkansson LK <
> stefan.lk.hakansson@ericsson.com>, public-media-capture@w3.org, Audio
> Working Group <public-audio@w3.org>
>
>
> Hello, and thanks for your input!
>
> I'm seriously in two minds about this - on one hand, it seems like
> functionality that is well worth having.
>
> On the other hand, it seems like a long list of things that could be of
> interest here, and I can easily envision considerable time passing while
> we discuss the details of each (for instance, if we expose the fact that
> an output is HDMI, we also expose the fact that it's either crypto
> capable or not crypto capable....)
>
> I think a lot of things can be addressed within the
> capabilities/constraints/settings model we've adopted for getUserMedia -
> one can define new constraints that get you the selectivity you want,
> one can call getCapabilities() to figure out what kind of device one
> has, one can use getSettings() to figure out what the current state of
> play is. If so (and if the TF keeps the "registry" approach for
> constraints), solving these problems can be as easy as authoring an
> add-on document called "additional audio capabilities and constraints".
>
> But I'm not sure if that will cover all your needs, or if this is the
> most elegant way of doing it - certainly some will make immediate note
> that the constraints mechanism isn't what they consider elegant.
>
> What do you see as the best way forward here - aim to address this
> later, or do we have parts of this problem that we *have* to address now?
>
>             Harald
>
>
> Den 21. april 2015 20:59, skrev Joe Berkovitz:
> > Hello Stefan,
> >
> >
> > Thank you for your recent solicitation of feedback to on the Media
> > Capture and Streams API, which I passed to the Web Audio Working Group.
> >
> >
> > The Web Audio WG so far has identified one key item that we would like
> > to see addressed. The MediaDeviceInfo result from enumerateDevices()
> > (
> http://www.w3.org/TR/2015/WD-mediacapture-streams-20150414/#idl-def-MediaDeviceInfo
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_TR_2015_WD-2Dmediacapture-2Dstreams-2D20150414_-23idl-2Ddef-2DMediaDeviceInfo&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=TljDhqSLOq4OTqBOCkRFi2iTNGDcdg0vbZ9A-vrOnlw&e=>
> )
> > lacks information that is typically available in the underlying OS
> > implementations that we think would be very helpful for implementations:
> >
> > __ __
> >
> > __·         __Channel count and configuration (Mono, Stereo, 5.1, 7.1,
> > etc…)____
> >
> > __·         __Physical Output (Headphone, Speaker, HDMI, …)____
> >
> > __·         __Latency (this matters a lot for gaming -- it will be very
> > low for on-board hardware, perhaps quite high for wireless audio
> > bridging like Apple TV)____
> >
> > __·         __Output capabilities (bitstream passthrough vs PCM –
> > relevant in digital media adapter cases (Chromecast, etc))____
> >
> >
> > It is perhaps sufficient from a user interface point of view to have a
> > string to display, but for a program to be able to either adapt to the
> > user selection or to guide and default the user selection, the above are
> > pretty important characteristics, at least in some use cases. Many if
> > not most of the host OSes that user agents run on expose these sorts of
> > output device characteristics. ____
> >
> >
> > Aside from the difficulty with enumerating devices, there is also
> > perhaps a need to make it possible for applications to query the set of
> > available devices with respect to the above
> > charateristics. MediaTrackConstraints and MediaTrackSettings do not
> > currently  include constraint attributes that map to items in the above
> > list. And even if they do, arriving at a practical goodness-of-fit
> > metric that can be generalized across a spectrum of audio apps may be
> > difficult.
> >
> >
> >
> > The same concerns apply to the set of input devices.__
> >
> > __ __
> >
> > Please let us know if this issue makes sense to the group and can be
> > addressed within the timeframe of the coming run-up to a Last Call WD.
> > We'd be happy to arrange some sort of inter-WG call to try to make
> > progress on this together.
> >
> >
> > Thank you!
> >
> >
> > Best regards,
> >
> >
> > Joe Berkovitz
> >
> > co-chair Web Audio WG
> >
> >
> > *Noteflight LLC*
> > Boston, Mass.
> > phone: +1 978 314 6271
> > www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=>
> <http://www.noteflight.com/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com_&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=TX3MVside5EU5bm_UNZyg2r1SdoBFsm-f8nP7K1k4Y8&e=>
> >
> > "Your music, everywhere"
> >
> > On Wed, Apr 15, 2015 at 1:31 AM, Stefan Håkansson LK
> > <stefan.lk.hakansson@ericsson.com
> > <mailto:stefan.lk.hakansson@ericsson.com>> wrote:
> >
> >     The WebRTC and Device APIs Working Groups request feedback on the
> Last
> >     Call Working Draft of Media Capture and Streams, a JavaScript API
> that
> >     enables access to cameras and microphones from Web browsers as well
> as
> >     control of the use of the data generated (e.g. rendering what a
> camera
> >     captures in a html video element):
> >     http://www.w3.org/TR/2015/WD-mediacapture-streams-20150414/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_TR_2015_WD-2Dmediacapture-2Dstreams-2D20150414_&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=iqvKvUbbXBvyilFPRoiU-moSntiBDqoGqKdbqREA2EY&e=>
> >
> >     The groups have identified the following other W3C Working Groups as
> >     likely sources of feedback:
> >
> >     - HTML Working Group, especially the HTML Media Task Force, as our
> API
> >     extends the HTMLMediaElement interface and defines a new type of
> media
> >     input via MediaStream
> >
> >     - WebApps Working Group, especially on the overall usage of Web IDL
> and
> >     the definition of error handling
> >     Audio Working Group, as the Web Audio API builds upon the MediaStream
> >     interface
> >
> >     - WAI Protocol and Formats Working Group, especially on the impact of
> >     the user consent dialog and the applicability of the indicators of
> >     device usage in assistive tools
> >
> >     - Web and TV Interest Group, as the manipulation of media input can
> be
> >     relevant to some of their use cases (e.g. glass to glass)
> >
> >     - Web App Security Working Group, especially on our links between
> >     secured origins and persistent permissions, and our current policy
> with
> >     regard to handling access to this "powerful feature"
> >
> >     - Web Security Interest Group, especially on our security
> considerations
> >     Privacy Interest Group, as access to camera and microphone has strong
> >     privacy implications
> >
> >     - Technical Architecture Group, for an overall review of the API,
> >     especially the introduction of the concept of a IANA registry-based
> >     constraints system, the use of promises, and our handling of
> persistent
> >     permissions
> >
> >     We naturally also welcome feedback from any other reviewers.
> >
> >     The end of last call review for this specification is set to May 15
> >     2015; should that deadline prove difficult to meet, please get in
> touch
> >     so that we can determine a new deadline for your group.
> >
> >     As indicated in the document, comments should be sent to the
> >     public-media-capture@w3.org <mailto:public-media-capture@w3.org>
> >     mailing list.
> >
> >     Thanks,
> >
> >     Frederick Hirsch, Device APIs Working Group Chair,
> >     Harald Alvestrand and Stefan Hakansson, WebRTC Working Group Chairs
> and
> >     Media Capture Task Force Chairs
> >
> >
> >
> >
> > --
> > .            .       .    .  . ...Joe
> >
> > *Joe Berkovitz*
> > President
> >
> > *Noteflight LLC*
> > Boston, Mass.
> > phone: +1 978 314 6271
> > www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=>
> <http://www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=>
> >
> > "Your music, everywhere"
>
>
>
>
>
> --
>
> .            .       .    .  . ...Joe
>
>
>
> *Joe Berkovitz*
>
> President
>
>
>
> *Noteflight LLC*
>
> 49R Day Street / Somerville, MA 02144 / USA
>
> phone: +1 978 314 6271
>
> www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=Ajygd3cU_M15NMeR6tSVYxwZiRtNw9yzWnTx0nK85QM&s=jXjlONf3ezJhUogvhWPTTov9Nkgv6NEMH3VU7EtbI5w&e=>
>
> "Your music, everywhere"
>
>
>
>
>
>
>
> --
>
> .            .       .    .  . ...Joe
>
>
>
> *Joe Berkovitz*
>
> President
>
>
>
> *Noteflight LLC*
>
> 49R Day Street / Somerville, MA 02144 / USA
>
> phone: +1 978 314 6271
>
> www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=A7v4sR8KByGJfQi7evt4CqIXowTuRgp3-n3YUTJ23eg&s=nx5xoRDlvECBLxTMg3j5OvvDclH_b-eKQ4hqJJ_Sitc&e=>
>
> "Your music, everywhere"
>
>
>
>
>
>
>
>
>
> --
>
> .            .       .    .  . ...Joe
>
>
>
> *Joe Berkovitz*
>
> President
>
>
>
> *Noteflight LLC*
>
> 49R Day Street / Somerville, MA 02144 / USA
>
> phone: +1 978 314 6271
>
> www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=A7v4sR8KByGJfQi7evt4CqIXowTuRgp3-n3YUTJ23eg&s=nx5xoRDlvECBLxTMg3j5OvvDclH_b-eKQ4hqJJ_Sitc&e=>
>
> "Your music, everywhere"
>
>
>
>
>
>
>
> --
>
> .            .       .    .  . ...Joe
>
>
>
> *Joe Berkovitz*
>
> President
>
>
>
> *Noteflight LLC*
>
> 49R Day Street / Somerville, MA 02144 / USA
>
> phone: +1 978 314 6271
>
> www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=A7v4sR8KByGJfQi7evt4CqIXowTuRgp3-n3YUTJ23eg&s=nx5xoRDlvECBLxTMg3j5OvvDclH_b-eKQ4hqJJ_Sitc&e=>
>
> "Your music, everywhere"
>
>
>
>
>
> --
>
> .            .       .    .  . ...Joe
>
>
>
> *Joe Berkovitz*
>
> President
>
>
>
> *Noteflight LLC*
>
> 49R Day Street / Somerville, MA 02144 / USA
>
> phone: +1 978 314 6271
>
> www.noteflight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.noteflight.com&d=AwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=qzKCNHFKJMzZBJ52at1DkA-_8TPxvcij-zS_VXs8c5A&m=A7v4sR8KByGJfQi7evt4CqIXowTuRgp3-n3YUTJ23eg&s=nx5xoRDlvECBLxTMg3j5OvvDclH_b-eKQ4hqJJ_Sitc&e=>
>
> "Your music, everywhere"
>
Received on Tuesday, 26 May 2015 17:53:06 UTC