TPAC 2020 Audio WG/CG meeting minutes from Raymond Toy on 2020-10-23 (public-audio-comgp@w3.org from October 2020)

From: Raymond Toy <rtoy@google.com>
Date: Fri, 23 Oct 2020 13:16:38 -0700
To: "public-audio@w3.org Group" <public-audio@w3.org>, public-audio-comgp@w3.org
Message-ID: <CAE3TgXHX0vRLWJMD16GT6ny-+dLpHJzDYVxESezwKwWAbiuFiQ@mail.gmail.com>

Thanks to everyone who attended the meetings, especially to all the CG
members for their patience as we went through the issues, exciting and
mundane.

Here are the minutes from all 10 meetings.

See also
https://docs.google.com/document/d/1l8yniQXgOUTNwEO7-I4YZBiW11-NrXz3L4zdNr156Io/edit?usp=sharing

TPAC 2020: Audio WG Minutes

Oct 12-15, 19-22
Agenda

Meeting agenda
<https://lists.w3.org/Archives/Public/public-audio/2020OctDec/0002.html>
MinutesOct 12Attendees

Christopher Needham, Jeff Switzer, Matthew Paradis, Raymond Toy, Hongchan
Choi, Paul Adenot, Chris Wilson, Takashi Toyoshima, Chris Lilley

Minutes

Introductions
-

Raymond Toy, Chrome, co-chair
-

Matt Paradis BBC, co-chair
-

Hongchan Choi, Chrome
-

Paul Adenot, Mozilla
-

Christopher Needham, BBC,
-

Jeff Switzer, CG member
-

Status of V1
-

rtoy@: Summarizes what Chris Lilley said last meeting
-

https://github.com/WebAudio/web-audio-api/issues/1933
-

rtoy@: we also have a dozen, but they are not controversial. We just
need to get them done.
-

padenot@: they are spec-technical, not the computer-techincal. :)
-

padenot@: The next round of security review should be okay, because
we already passed it twice in the past. So not a blocker? but we’ll see.
-

matt@: the impact on the test suite when we advance to REC? when some
impl fail, and others pass?
-

padenot@: if there’s a clear intent to implement (from all vendors)
then we are fine.
-

hoch@: can W3C do something when the inconsistencies are noticeable?
-

all@: ¯\_(ツ)_/¯
-

cwiso@: hello
-

rtoy@: https://github.com/WebAudio/web-audio-api/issues/2248
-

Web MIDI (cwilso@, toyoshim@)
-

Agenda
<https://docs.google.com/document/d/1wqdYBCy3p_r3lx2CqU5JoC-HM37CFY0FqMxcxRdEdsA/edit>
by toyoshim@
-

toyoshim@: Implementation updates from Chrome Web MIDI
-

we plan to prompt the permission dialog always.
-

updated the Windows MIDI backend to the recent one (work in
progress)
-

padenot@: we have all the codes! working and tested. No shipping date
yet.
-

back-end for Mac OS -> cross platform MIDI library
-

hongchan@; Perhaps notes and CC can be shipped first?
-

padenot@: positive about it, but no solid timeframe yet
-

toyoshim@: do we need to change the spec? anything to improve?
-

padenot@: no. fine as it is.
-

cwilso@: we don’t need to worry about V2 until there’s an actual
device supports it. The biggest unsolved issue in V1 spec is the
“backpressure”. that can be solved with the stream.
-

hongchan@: so the stream is the solution?
-

(more thoughts on backpressure from cwilso@)
-

cwilso@: midi message as a user gesture => not a good idea (any
connected Midi devices can send a message)
-

toyoshim@: new issue
https://github.com/WebAudio/web-midi-api/issues/215
-

hongchan@: in worker instead?
https://github.com/WebAudio/web-midi-api/issues/99
-

padenot@, hongchan@: SAB is better in this case
-

/* hongchan@ couldn’t scribe… */
-

// traiging/resolving the issues on the tracker…
-

Additional notes from toyoshim@:
-

https://docs.google.com/document/d/1wqdYBCy3p_r3lx2CqU5JoC-HM37CFY0FqMxcxRdEdsA/edit?usp=sharing
-

Updates:
-

Implementation updates for Chrome
-

No visible progress due to lack of engineering resource now,
but there are several internal design changes to be
aligned with current
chrome design
-

Remaining works for the current latest spec
-

Permission prompt for non-sysex requests
-

MIDIOutput.clear()
-

Windows backend API change from Win32 MMAPI to WinRT does
not happen yet, and still behind a flag
-

Can not obtain equivalent devices names that reflects
product names via the WinRT API
-

Implementation updates for Firefox
-

Sysex is only security concern
-

Keep working on, but no timeline for now
-

May have feedback on the spec, but will file issues when
things matter
-

Spec
-

Remaining works for V1 (
https://github.com/WebAudio/web-midi-api/issues)
-

Major topics
-

Back pressure, probably using Streams
-

Handling MIDI messages as a user gesture -> not good ideas
-

Visited V1 targeting open issues, and took actions
-

Revisit the goal or the next step
-

MIDI 2.0 (
https://www.midi.org/specifications-old/category/midi-2-0-specifications-v1-1
)
-

Relevant issues
-

https://github.com/WebAudio/web-midi-api/issues/211
-

https://bugs.chromium.org/p/chromium/issues/detail?id=1047087
-

https://discourse.wicg.io/t/web-midi-api-support-for-midi-2-0/4208
-

Conclusion: It isn’t mature enough to discuss here
-

Meeting Adjourned (10:54AM)

Oct 13Attendees

Peng Liu, Philippe Milot, Philippe Le Hegaret, Jeff Switzer, Hongchan Choi,
Matt Paradis, Raymond Toy, Chris Lillley, Jack Schaedler, Paul Adenot,
Vainateya Koratkar

Minutes

9:07: Meeting started
-

Introduction
-

PLH: W3C project management
-

P Milot: Audio Kinetic
-

Peng Liu: WebKit media Apple
-

Matt: BBC, co-chair
-

Jeff Switzer: dev, CG member
-

Agenda
-

Spec V2 logistics
-

rtoy@: i had a PR for v2 spec, a boilerplate
-

rtoy@: V2 - it will be little changes, how do we work on that?
-

rtoy@: noise generator is pretty worked out
-

matt@: going from V1 to V2, including changes in V2 new nodes how
do we make it neat/tidy way?
-

rtoy@: AudioParam power law - V1 addendum?
-

chrislilly@: if make a major change, we can copy over to V2.
-

rtoy@: new phase oscillator? we copy the entire oscillator section?
-

clilley@: yeah we should copy over.
-

rtoy@: once we go to REC, we can copy the entire V1 to v2.
-

cllilly@:
-

hongchan@: Will file a new issue on how to write V2.
-

Priorities for V2
-

Hongchan presents survey results
<https://meet.google.com/linkredirect?authuser=0&dest=https%3A%2F%2Fdocs.google.com%2Fpresentation%2Fd%2F1DNjlh_JwjfwDzoULAUx5wUj2Igrx-eUbZ2ZHltLGOZo%2Fedit%3Fusp%3Dsharing>
..
-

padenot@: Matches my expectations
-

Jack: Wonders if render size responses would be different if the
question were asked differently to indicate that Android
devices can only
use 50% of the CPU with the current 128 frame size.
-

Dev tool support
-

padenot@: ECMAscript has added finalizers allowing implementing
large parts of the dev tool support. The other parts are
probably doable
by writing a cross-browser extension. Firefox usage was
rather small, but
those users really liked it a lot.
-

chrislilley@: asks about DTS streaming.
-

hongchan@: Not really webaudio
-

padenot@: HTML audio stream seems the right place
-

matt@: Let’s follow the proposed prioritization.
-

Output device selection
-

WASM integration
-

Output device change
-

Configure audio device
-

CPU usage
-

Output device selection
-

padenot@: Long history for this. See the following for some
background:
-

https://github.com/w3c/mediacapture-output/issues/50
-

https://github.com/w3c/mediacapture-output/issues/49 Supported
today when device output changes
-

https://github.com/w3c/mediacapture-output/issues/46
-

Philippe Milot: What is needed to move this forward?
-

padenot@: Writing spec and implementing. Some complications with
multiple AudioContexts connected by a media stream.
-

hongchan@: What’s the history here?
-

Paul: Was told this is where output device selection was supposed
to be done.
-

Paul: Permissions and enumeration handled for us with mediacapture.
-

Hongchan: Not much configurability? Can’t select channel counts,
etc.
-

Hongchan: Also
https://w3c.github.io/mediacapture-output/#mediadevices-extensions
-

Paul: There is a move in webrtc to expose less things that can be
fingerprinted. Old style is enumeration with list of devices
and IDs. Now
you request capability and get one device that matches.
-

Philippe Milot: Seems good for our use case as long as the info
contains all of the desired info.
-

Hongchan: need to extend this
-

Paul: We can do a constraint pattern to select the device.
-

Hongchan: Related question: Can you determine what kind of output
is connected? LIke headset or speakers. Useful for games.
-

Philippe Milot: Yes, it’s useful. Platform dependent but a
boolean is good enough to say it’s headphones or not. See
https://github.com/WebAudio/web-audio-api-v2/issues/89
-

Hongchan: This is relevant
https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/ondevicechange,
but we need to provide more information about the device.
-

Paul: Also relevant:
https://github.com/w3c/mediacapture-main/issues/730
-

Hongchan; For configurability, what else do we need to request
besides sample rate and number of channels.
-

Phillipe Milot: Wyse would want to present a list to user to
select the desired output device. Then use the device to get the
parameters.
-

Paul: Constraints don’t have to be exact.
-

Hongchan: How to use channel mask?
-

Philippe Milot: Mostly to determine where the LFE channel is.
Typically WAV format. Complicated code to get vorbis and
opus(?) channels
to WAV because the ordering isn’t the same for the codecs.
-

Paul: We use SMPTE, it’s in our spec
-

Hongchan: How do we work on this? In Audio WG or in MediaCapture?
-

Paul: Ideal to use MediaCapture to support what we need.
-

Matt: Do we want this?
-

Hongchan: We do want this but might want to piggyback on
MediaCapture
-

Paul: Same for us.
-

Matt: Ready to move on to the next item on the list.

Oct 14Attendees

Jack Schaedler, Philippe Milot, Raymond Toy, Hongchan Choi, Paul Adenot,
Matt Paradis

Agenda

Use WASM within Audio Worklet more efficiently. (issue
<https://github.com/WebAudio/web-audio-api-v2/issues/4>, issue
<https://github.com/WebAudio/web-audio-api-v2/issues/5>)
-

https://github.com/whatwg/html/pull/6056
-

Programmatically inspect the audio rendering CPU usage in JS. (issue
<https://github.com/WebAudio/web-audio-api-v2/issues/40>)
-

More AudioNodes: Noise generator, Noise gate, Expander, and new
Oscillators

Minutes

WASM with AudioWorklet
-

Hongchan:
https://github.com/WebAudio/web-audio-api-v2/issues/5#issuecomment-673589793
-

Raymond: Yes, knowing the cost would be beneficial. I vaguely
remember Jack saying he was interested in doing this. My guess
is the cost
of copying is small, but if the worklet is a gain node, it’s a
significant
fraction.
-

Paul: Lots of non-linearity. Depends on cache effects depending on
the memory used. Will take some care to measure to get
meaningful results.
-

Philippe: Closely related to BYOB issue?
-

Paul: Yes.
-

Hongchan: Can we do this without help from WASM people.
-

Paul: Yes, but we can use functions instead of class.
-

Hongchan: How does the worklet know about these functions?
-

Paul: Can use C-style to pass state and functions (pointers) to
WASM. Need to ensure minimal overhead. WASM can only call functions.
-

Raymond: Can developers already do this?
-

Paul: implement register function to register the process function
with state and function.
-

Hongchan: Sounds like animation worklet.
-

Paul: Yes, but kind of the opposite.
-

[Paul updates the issue #5]
-

Paul: May want to have a simple example with this idea to start
with. Will write one up later.
-

Hongchan: How is #4 related? It’s about registering memory
-

Paul: Having #4 and #5 is best, but just one is ok.
-

Hongchan: Memory needs to be owned by the node, not wasm. Perhaps
done in the constructor?
-

[Hongchan summarizes discussion in #4] - link
<https://github.com/WebAudio/web-audio-api-v2/issues/4#issuecomment-708519581>
-

Hongchan: What happens if the channel count changes requiring more
memory
-

Paul: Could have a callback saying more memory is needed. Lots of
memory could be allocated and a pointer to the block to be used
by the node.
-

Hongchan: Will write a simple example for #4.
-

Worklet
-

Worklets are being moved to the HTML standard
<https://github.com/whatwg/html/pull/6056>
-

Paul: This is a welcome change.
-

Hongchan: Just need to be aware of the upcoming changes to make sure
AudioWorklets are fine.
-

CPU usage
-

Hongchan: Paul says to allow performance.now() in the worklet.
-

Hongchan: How does this apply to the entire graph?
-

Paul: It doesn’t really.
-

Hongchan: Would like this for the entire graph.
-

Hongchan: Is performance.now() heavy?
-

Paul: Can be fast, depending on the OS. (Very cheap on mac. More
complicated on Win). Probably used during development, not production.
-

Raymond: Is performance.now() resolution limited?
-

Paul: Yes. Happens on non-isolated machines with spectre issues. Or
if anti-fingerprinting enabled (in Firefox).
-

Raymond: Probably not a show-stopper. Just call performance.now()
every 10 or 100 renders.
-

Paul: Let’s proceed as if performance.now() is useful for audio
profiling.
-

Raymond: Still want to add API for the entire graph?
-

Hongchan: Yes. Something like AudioContext.foo to get usage.
-

Jack: I think most devs who are asking for this, are mainly wanting
to be able to answer the question, "Should I turn off the Reverb
on my app
because I'm on some mobile device that is having trouble keeping up?"
-

Raymond: Does providing the CPU usage for Webaudio work?
-

Jack: Yes. Something more “chunky” is totally fine.
-

Raymond: Already designed most of the API
-

Hongchan: Want to talk to privacy people
-

Paul: Definitely, since it says something about the machine.
-

Hongchan: Come up with a rough API design.
-

Paul: Date.now() does work in an AudioWorklet.
-

Render size
-

Philippe: Proposes discussing this tomorrow
-

[General agreement]
-

Audio Nodes
-

Noisegate/Expander (issue
<https://github.com/WebAudio/web-audio-api-v2/issues/12>)
-

Raymond: Paul mentioned in the issue that a prototype would be
really useful.
-

Echo cancellation?
-

Hongchan: Could we implement echo cancellation with webaudio?
-

Paul: Should work. But capturing all other audio is difficult or
impossible (Mac).
-

Raymond: Yeah, exposing other audio to the web seems scary.
Capturing the mic input and audio output from a tab seems ok and echo
cancellation should work. Just can’t cancel other audio from
other sources.
-

Raymond: Maybe the only thing missing is an API to allow access to
all other audio sources. I never have any other audio when on a call.
-

Adjourned. We’ll start with render size tomorrow.

Oct 15Agenda

Select the render block size of the Web Audio renderer (issue
<https://github.com/WebAudio/web-audio-api-v2/issues/13>)

Attendees

Philippe Milot, Raymond Toy, Hongchan Choi, Paul Adenot, Ruth John

Minutes

Render size
-

Paul: There’s agreement to do this. Do we want to discuss the API?
-

Raymond: Yes.
-

Paul: Default is 128. There should be a way to ask for optimal. And
to be able to specify a number.
-

Paul: Is the number a hint?
-

Raymond: Yes. I want browsers not to have to support all possible
values. The spec should require powers of two, and leave the rest to the
browser
-

Philippe: Can the spec mandate certain values?
-

Paul: Yes. Multiples of 128 should not be a problem.
-

Raymond: What about 384? Chrome currently wants convolvers to use a
power of 2. Might work by buffering the data for the convolvers
to operate
only on power-of-two sizes. Makes the convolver processing bursty.
-

Raymond: On windows 440 size could be done with 512, losing about 15%
CPU.
-

Paul: Yeah. But definitely a problem on Android.
-

Paul: Pixel (4?) is 96.
-

Raymond: Would powers of two be good enough?
-

Jack: Yes
-

Raymond: Proposal: It’s a hint; only powers of two are required; an
attribute is needed on the context to indicate the actual size.
-

[Updated issue with proposal]
-

Raymond: Paul’s last comment about device changes
-

Paul: Yeah, devices changes can be detected.
-

Raymond: But device changes don’t affect these;
-

Paul: Yeah. Things should continue as is with buffering/resampling.
Devs need to redo the graph if needed.
-

Raymond: Paul: what does “being able to know the right size”
-

Paul: Something in media devices to get the HW size. But not really
needed now. Just request the optimum size.
-

Raymond: Do we want to expose the HW size in addition to the actual
size?
-

Paul: No, not necessary
-

Philippe: Not necessary
-

Other P1 issues (issues
<https://github.com/WebAudio/web-audio-api-v2/issues?q=is%3Aopen+is%3Aissue+label%3Apriority-1>
)
-

NoiseGate (https://github.com/WebAudio/web-audio-api-v2/issues/12)
-

taking the modular approach: build envelop follower
-

Raymond: Is providing envelope follower enough to build everything
with other native nodes?
-

Paul./ChrisL: Yes. Just need gain, waveshaper, and delay.
-

Paul: What’s the latest in subclassing AudioNode?
-

Hongchan: Would be great to have. Have seen people do this.
AudioWorkletNodes support this and it works.
-

Raymond: My guess is that people who want this want a node to
throw in and not have to build it themselves.
-

Hongchan: Curious about this approach. Will it be fast and
efficient?
-

Paul: Could mix and match with native nodes and worklet.
-

Raymond: My take is that either we should provide the node (no
envelope follower) or let devs create their own with a
worklet. Providing
a follower isn’t enough.
-

Raymond: What’s the conclusion here?
-

Paul: It’s easy with an envelope follower. Just a couple of lines.
-

Raymond: I’m ok with that if the spec has examples showing how to
do a nice noisegate/expander.
-

Raymond: Conclusion is that community input is really important?
-

Playback position
<https://github.com/WebAudio/web-audio-api-v2/issues/26>
-

Paul: I like the term “position” instead of “time”.
-

Raymond: Works for me.
-

Jack: Just need to update once per frame (animation frame)
-

Paul: Let me write a proposal in the issue.
-

Raymond: Looks good.
-

Change outputChannelCount dynamically
<https://github.com/WebAudio/web-audio-api-v2/issues/37>
-

Paul: Firefox only allows changes between renders
-

Raymond: Chrome is the same, except the may not happen in the next
render because it can’t get the locks needed to make the change.
-

Raymond: Use case with a convolver is a good example. Mono input
with mono response is mono. But change the response to
stereo, and the
output is stereo. Can’t do that currently with an AudioWorklet.
-

Paul: Let’s provide an API to set the output channel count. But
also need a way to unset this change.
-

[Paul updates the issue]
-

Handling of output parameter for process()
<https://github.com/WebAudio/web-audio-api-v2/issues/42>
-

Raymond: The original issue is no longer relevant because the
parameters are frozen now. (Discussion moved into different directions)
-

Paul: This is an edge case that makes things hard.
-

Paul: Example: Delay node with 4 s delay, and a stereo input that
changes to mono. The delay needs to preserve stereo for 4 more sec.
-

Raymond: Yeah, I don’t know how to handle that in a worklet.
Could have process send a message to the main thread to
request changing
the channel count after 4 sec. Not terrible but ok. (This
could have been
handled nicely if we had made process() return the output
data instead of
passing in the output arrays. Too late for that.)
-

Adjourned. Continue next time with handling of output parameter.

Oct 19Attendees

Jack Schaedler, Lora Friedenthal, Raymond Toy, Chris Lilley, Paul Adenot,
Jer Noble, Peng Liu, Matt Paradis

Minutes

Handling of output parameter for process()
<https://github.com/WebAudio/web-audio-api-v2/issues/42>
-

Raymond: Reiterates we should close this because this isn’t possible
anymore. It should be taken up in #37.
-

Paul: Way forward is changing the channel in the RT thread is
out-of-scope so you can’t change the number of output channels
in the same
process call. One way out is if we move to BYOB to change the channel
output.
-

Raymond: Yes, but that’s really part of #37 and the BYOB issue.
-

Paul: Yes.
-

[Raymond updates and closes #42]
-

[Paul updates #37]
-

Update the README.md
<https://github.com/WebAudio/web-audio-api-v2/issues/75>
-

[Paul updates the issue]
-

High res timer in AudioWorkletGlobalScope
<https://github.com/WebAudio/web-audio-api-v2/issues/77>
-

Paul: Need performance.now() and cpu load factor (issue #). Should
solve the problems with devs knowing how to profile and adjust the graph
appropriately.
-

Jer: [Mic not working]. Do you need a specific clock or whether a
monotonically increasing counter is needed.
-

Paul: It’s complicated. Talked to a lot more people and they want to
know load measure to see if they’re about to glitch. For this
you need to
have some real-time value. The engine could compute the load and expose
that.
-

Jer: Could give quantized load factor, say 5% level. Not sure about
exposing perf.now() given spectre. Maybe ok with enough cpu mitigations.
-

Jack: I think for the real nitty gritty profiling, using the browser
tools is actually preferable to doing something with
performance.now(). For
the case of computing load and responding on a machine you might not have
control over, I think most devs would be fine with a really coarse
quantization of the load, ~10% or so.
-

Raymond: Agrees that you need dev tools for detailed profiling.
-

[Paul summarizes in the issue]
-

Update from Apple
-

Jer: Last release concentrated more broadly on supporting major
websites. Major pain point was the lack of AudioWorklet. This is now
available (or will be soon). Includes unprefixing. Safaris 114 is the
current version, and worklet should be in the next one. Should be
cross-platform. iOS will have it in the next full release.
-

Jack: yeah, hard to overstate how exciting this is!
-

Input only/muted AudioContext
<https://github.com/WebAudio/web-audio-api-v2/issues/83>
-

Jer: For analysis?
-

Raymond: Yes.
-

Jer: Use case: users on iOS devices with hearing aids via
bluetooth. Media isn’t heard.
-

Paul: We do something like this.
-

Paul: Start in input-only mode and change output device to hear
audio, glitch-free. This worked in Firefox OS.
-

Raymond: That seems really complicated.
-

Jer: In context of the issue, this seems really just analysis only.
-

Raymond: Yes. WebRTC asked us about this exact ability some time ago
to bypass the user-gesture when only doing analysis.
-

Raymond: This particular use case seems straightforward, but the
other cases Paul mentioned about switching outputs and enabling
audio seem
really complicated.
-

Paul: Yes. I don’t have a solution.
-

[Other discussion that I didn’t capture]
-

Raymond: This use-case is simple. The other use-cases should perhaps
be a different issue?
-

Paul; Need more input on uses cases and CG help here.
-

[Paul updates issue]
-

Priority 1 issues are done; moving on to priority 2
-

Allow OfflineAudioContext to render smaller chunks repeatedly
<https://github.com/WebAudio/web-audio-api-v2/issues/6> (issue #6)
and incremental
delivery of gargantuan data from OfflineAudioContext
<https://github.com/WebAudio/web-audio-api-v2/issues/66> (issue #66)
-

Raymond: This (and #66) came up many TPACs ago and Jer’s suggestion
about calling startRendering() again would work. Question is how to
specify how much data to render.
-

Paul: Oftentimes we think streams would be the solution, but
sometimes it’s not.
-

Jack: I like where Raymond is going with the idea there. The
intention, "Give me the next N samples" from the context is
pretty nice for
many reasons.
-

Jer: What’s the proposal? startRendering(nsamples)?
-

Paul: That’s one approach. Streaming would work.
-

Jer: Use case involves raw audio or encoded? Presumably encoded to
be uploaded somewhere?
-

Jer: Minimal AP change is startRendering(n), allowing startRendering
to be called more than once.
-

Paul: I can see using streams
-

Jer: Yes, and you can write your own stream handler.
-

Paul: WebRTC is moving towards streams. So is WebCodecs. Very
elegant to be able to interact with this.
-

Raymond: Streams are fine with me; just need an API.
-

Paul: Would be nice to see what #66 would look like with a stream.
-

Paul: Need to re-read streams section.
-

Raymond: Do we want to make a proposal?
-

Jack: Callback with a buffer is just so simple and easy, but I
definitely see the appeal of the stream based approach
-

Raymond: Proposal: close #6 in favor of #66 and make sure #66 handles
the termination case.
-

[Paul writes summary in issue]
-

[Raymond updates #6 and closes it]

Oct 20Attendees

Jack Schaedler, Paul Adenot, Raymond Toy, Hongchan Choi
Minutes

Storing AudioBuffers in different bit depths
<https://github.com/WebAudio/web-audio-api-v2/issues/11>
-

Paul: Not sure what to do here. Streaming decode is already
available. Do people really need in-memory assets? Probably.
-

Raymond: I think people maybe just want decodeAudioData to have
16-bit data, all done transparently, like Firefox does today.
-

Paul: I’m looking up someone’s blog about free algorithms for
in-memory compression of audio.
https://www.bitsnbites.eu/hiqh-quality-dpcm-attempts/
-

Paul: I’m envisioning some method to allow compressing audio
(decodeAudioData) using DDPCM.
-

Raymond: But it’s lossy.
-

Paul: I wonder how it compares to mp3.
-

Raymond: What about FLAC? LIcense is free
-

Paul: It’s pretty complex. Indexing into the file to get the right
data is expensive. And looping makes it hard.
-

Raymond: Yeah, looping is a problem.
-

Raymond: No objections as long as users have to ask for lossy
compression
-

Paul: Of course.
-

Raymond: Not sure where to go from here. We’ve asked for proposals
on what this should look like.
-

Raymond: We all agree this is useful; we just don’t know what it
would look like and what it should do.
-

[Paul writes summary]
-

Real-time pitch adjustment
<https://github.com/WebAudio/web-audio-api-v2/issues/14>
-

Paul: I’ve heard that students have implemented pitch-shift in
AudioWorklet. But failed to ask where the code is.
-

https://github.com/olvb/phaze/ This is pitch shifter, not time
shifter.
-

Raymond: Doesn’t seem to be all that big
-

Paul: Yeah, it’s really tiny.
-

Jack: I'd be very interested to try this implementation out!
-

Hongchan: Is this a source node or processing node?
-

Jack: This definitely feels like a case where you'd want to prototype
a bunch of options using AW, and work through all the kinks. Consider
playing backwards, etc. A source would perhaps make more sense
-

Jack: Crossfading and looping cleanly might also require having
access to the source.
-

Jack: Yeah, for a pitch-shifter, or a paulstretch kind of thing, a
node is fine.
-

Jack: However, people are going to want to build things like warped
clips in Live/Bitwig, and that's going to be tricky with a node.
-

Raymond: I guess the next step is to get the example.
-

[Raymond summarizes in issue]
-

Worker support for BaseAudioContext
<https://github.com/WebAudio/web-audio-api-v2/issues/16>
-

Jack: The pain of scheduling contesting with the main thread is very
real :)
-

Hongchan: Seems really useful, but how useful would it be without
media streams and media elements. We’ll lose access to getUserMedia.
-

Paul: Found the proposal from Intel and Mozilla:
https://github.com/w3c/mediacapture-worker/ (draft
<https://w3c.github.io/mediacapture-worker/>)
-

Hongchan: Maybe split this into two parts: AudioContext in worker,
and media stream in worker?
-

Paul: Maybe we can transfer the streams? [Pings colleague on this,
asking for info on this].
-

Paul: Splitting this in two is fine. But what’s missing is
use-cases. There are people who care about this who don’t need media
stream. Sounds like a good plan to split.
-

Raymond: Someone want to summarize this and create the new issue?
-

Paul: Yes, I’ll do it.
-

Customizable windowing function for Analyser
<https://github.com/WebAudio/web-audio-api-v2/issues/19>
-

Raymond: I think we have the API designed, but need to decide what
happens when you switch the window type. I find it unlikely people would
want to change it after construction. Maybe not allow changes?
-

Jack: The only case I can imagine where that's important is someone
making a DSP tutorial or something where they want to compare
the effect of
the windowing functions. They can work around that by just making N nodes
-

ConvolverNode limited to two channels
<https://github.com/WebAudio/web-audio-api-v2/issues/20>
-

Raymond: Mostly historical. Not sure how to handle 4-channel
responses.
-

Raymond: Maybe add option to specify response is for individual
channels, no matrixing operations. Number of outputs is the number of
responses.
-

Raymond: Then only need to describe how to upmix the inputs.
-

Raymond: But do we really want to do this?
-

Paul: One of those things where if you’re not doing complex surround
sound, you don’t need it.
-

Jack: Yotam Mann would be good to talk to Yotam uses WebAudio for big
(32) channel student projects
-

Raymond: Yes, we should ask him.
-

[Raymond updates issue]
-

loadHRTFDatabase for SpatialPanner
<https://github.com/WebAudio/web-audio-api-v2/issues/21>
-

Paul: Talk to Matt about this.
-

Raymond: Let’s assign to Matt to get more information on what’s
needed here.
-

[Raymond updates issue]
-

Configurable sample rate quality
<https://github.com/WebAudio/web-audio-api-v2/issues/25>
-

Paul: What is the best way to measure this?
-

Raymond: The quality?
-

Paul: Was told that high quality with SIMD is faster than linear.
But didn’t verify the claim.
-

Raymond: Seems hard to imagine, but maybe.
-

Raymond: I see it useful for final output of AudioContext
-

Paul: Firefox does it carefully for ABSN, hiding the latency of the
resampler.
-

Jack: No, but I could make one if you want something similar to the
karplus tester thing if you want.
-

Raymond: What do you want to do?
-

Paul: To understand the cost of resampling.
-

Jack: I could just make a page with tons of buffers playing back in
an amusing way
-

Paul: I’ll get some data on this.

Oct 21Attendees

Paul Adenot, Raymond Toy, Hongchan Choi, Jack Schaedler, Matt Paradis,
Chris Lilley

Minutes

Configurable sample rate
<https://github.com/WebAudio/web-audio-api-v2/issues/25> (cont’d)
-

Raymond: Is there more to say about this?
-

Paul: This is really about final output?
-

Raymond: Mostly
-

Paul: Took a look and output resampling on Windows is different
between voice and general audio. In the latter, we do it more
carefully.
Output latency is about 70 frames (depends on sample rates).
-

Raymond: 70 frames is pretty good.
-

Hongchan: This should be configurable at runtime
-

Paul: I was testing yesterday with playback rate between 0 and 1
exclusive. Switched between high quality resampler and linear
interpolation. (High quality is very optimized with SIMD, etc.) No
results yet.
-

Paul: Could be playing back 8 kHz audio, low quality source, so low
quality resampler is not terrible.
-

Hongchan: How important is this?
-

Jack: I’m interested in any config settings to trade capacity for
quality. Especially important once render capacity is available.
-

Hongchan: Maybe start with final output
-

Paul: Devs may choose quality depending on asset. Long background
audio may not need high quality.
-

Hongchan: What’s the next step?
-

Paul: Output should be done. The rest could be important with
significant savings.
-

Raymond: Changing quality in ABSN while running would sound funny
-

Paul: Jack, how would you do this? Recreating the source?
-

Jack: That would work, but we might just reload the whole page
-

Paul: If we go per node, static could be enough.
-

Paul: Could be ABSN option
-

Raymond: What does FF use for resampling when playback rate is not 1?
-

Paul: Use the resampler from Opus, formerly from speex.
-

[Paul updates issue]
-

Setting convolution buffer asynchronously
<https://github.com/WebAudio/web-audio-api-v2/issues/28>
-

Hongchan: We should just do it? HRTF should be a separate issue
-

Paul: Yes, let’s do convolver here.
-

Raymond: Do we deprecate the buffer setter? Of course it won’t go
away.
-

Paul: I was thinking about just this the other day.
-

[Paul updates issue]
-

Add method to cancel ABSN.stop()
<https://github.com/WebAudio/web-audio-api-v2/issues/30>
-

Raymond: Calling stop with a large number works for me. Or maybe
with a negative number?
-

Raymond: But either way, the API would be weird to the OP.
-

Hongchan: No strong opinions; never encountered this situation.
-

Hongchan: Is this really priority-2?
-

Raymond: It was decided at the last F2F meeting; I don’t remember the
reasoning.
-

Raymond: I’d prefer not to add an API for this; stop(huge) works.
-

Raymond: What should we do?
-

Jack: Agree that it’s not priority-2.
-

[No conclusion]
-

When does stop(time) stop?
<https://github.com/WebAudio/web-audio-api-v2/issues/38>
-

Raymond: Since Chrome and Firefox do the same thing already, we just
need to make it clearer in the spec when stop actually happens. Not sure
what to say in the spec though.
-

Paul: What Karl says is right, but would prefer the way things are now
-

Raymond: Yes, Karl is right, but having audio before start and after
stop would be weird to developers.
-

[Raymond updates issue]
-

Use SharedArrayBuffer for getChannelData
<https://github.com/WebAudio/web-audio-api-v2/issues/39>
-

Chris: Not sure what the primary concern is.
-

Paul: The buffer could be read from a different thread while it’s
being modified. Not something we want.
-

Jack: We ran into this same issue and used WASM to work around this
issue. We had a WebWorker and doing the copy. Conceptually want a const
ref to pass the data around. It’s a real problem.
-

Paul: It’s a bit of a problem. It’s another way to get shared
memory. We need to be careful about what to do here, especially when it’s
not available.
-

Jack: Seems like the most common case is loading a big file and
drawing a waveform.
-

Paul: Agree, but we want to be careful about enabling shared memory.
-

Raymond: Would having a sharedArrayBuffer have helped you Jack?
-

Jack: If you can get the data to a webworker without a copy, that
would have helped us. Copying the data took more time than the
analysis on
the worker.
-

Paul: One of those things where I wish there was a way to freeze a
portion of memory.
-

Raymond: What should we say? Further study?
-

Paul: I think it’s fine.
-

Raymond: Can you write up a summary?
-

[Paul updates issue]
-

Informing AudioWorklet if output is not connected
<https://github.com/WebAudio/web-audio-api-v2/issues/41>
-

[Raymond summarizes the issue]
-

Raymond: Could be a useful optimization so worklet doesn’t have to
compute the output if the output isn’t connected.
-

Paul: Could be useful. Are there precedents in other frameworks?
-

Paul: What would it look like?
-

Paul: Don’t think changing the channel count is the right way.
-

Paul: People probably don’t check the output
-

Raymond: Yes, this will definitely break things
-

Paul: It’s useful though.
-

Raymond: No one is asking for this except me, and I don’t write
things.
-

Paul: Maybe ask people.
-

[Paul updates issue]
-

Support Q and S for shelf filters
<https://github.com/WebAudio/web-audio-api-v2/issues/67>
-

[Raymond links to demo
<https://rtoy.github.io/webaudio-hacks/more/biquad/biquad-coef.html>]
-

Raymond: Demos shows that we can’t make them backward compatible.
Need new types.
-

Paul: Yeah, just get a name and specify it.
-

[Raymond updates issue]

Oct 22Attendees

Lora Friedenthal, Jack Schaedler, Matthew Paradis, Raymond Toy, Hongchan
Choi, Paul Adenot
Minutes

Remove compileStreaming from AudioWorkletGlobalScope.WebAssembly
<https://github.com/WebAudio/web-audio-api-v2/issues/79>
-

Hongchan: Does compileStreaming involve multi-threading?
-

Paul: It works with fetch to start a streaming compile, or anything
else that can be streamed.
-

Paul: is Response only from fetch? Yes.
-

Hongchan: Seems quite useful.
-

Paul: But we don’t want to compile on the audio thread.
-

Hongchan: But this is a whole package coming from WASM.
-

Paul: That’s what I was asking but I don’t know if we can. Just need
to find the right person to ask the question.
-

Hongchan: [Action item to ask]
-

Hongchan: If the answer is no, what should we do?
-

Paul: Well it exists but is basically useless.
-

Paul: I guess we just need an answer from people who know.
-

[Paul updates issue]
-

Multithreading <https://github.com/WebAudio/web-audio-api-v2/issues/85>
-

Paul: Any new developments on this?
-

Hongchan: What about comment on Jun 13 (wytrych)
<https://github.com/WebAudio/web-audio-api-v2/issues/85#issuecomment-643620580>
-

Raymond: I think he means two contexts get separate threads in Chrome.
-

Paul: Firefox is thinking of changing its implementation to have
separate threads.
-

Hongchan: This is about multiple threads for all processing?
-

Hongchan: Seems interesting, but would need a rewrite. May take
several years.
-

Raymond: But would we expose this to the user?
-

Hongchan: Problem is unsynced clocks and lack of single clock.
-

Hongchan: Could be up to the UA.
-

Hongchan; Multiple pthreads in WASM with workers solves this.
-

Paul: Then we’re back to the problem of thread priorities.
-

Raymond: Can’t this be done inside a worklet?
-

Hongchan: I think so. Oh. Not in chrome. Pthreads not supported.
-

Paul: Probably don’t want to allow this either.
-

Hongchan; Ask developers if they want multiple threads of nodes, etc.
-

Paul: I linked the mailing list with more info.
-

Jack: This is also the use case I imagine most devs will want
addressed.
-

Hongchan: Even if you could spawn pthreads, they won’t be realtime.
-

Paul: That’s why I suggested adding a flag to say audio contexts run
on a separate thread. Basically opt-in to multiple threads.
-

Hongchan: Needs more clarification from devs. Insufficient detail in
the issue.
-

Paul: I think in the end, the F2F comment still holds, since Chrome
does this.
-

Jack: This topic is really complicated. I’m going to go back do some
homework, then write into the comments outlining the exact use cases that
are most likely.
-

Hongchan: I think the main point is "When running multiple contexts
in parallel we of course run into problems like unsynced clocks and the
lack of a single sink so it's impossible to have one master gain for
example"
-

Jack: I think the main/common desire is this: "I have a big WASM
app... I do some analysis of my custom audio graph, and realize
that I can
(in theory) render two portions of my graph in parallel... In a
DAW context
this might be two 'tracks' which don't interact in any way. If I could
create two threads from my worklet, that would be helpful." But
as you say,
if I can't create threads at the right priority and everything... then
-

Hongchan: If this is an issue with pthreads and WASM, then this is a
problem to be solved in WASM.
-

Jack: +1
-

Paul: Firefox allows 200 ms in a worklet (instead of the normal 3
ms). Then worklet errors out.
-

Hongchan: Want to ask WASM people if it’s possible to enable realtime
pthread.
-

Paul: There are native apps that allow this.
-

Hongchan: I’ll ask WASM people about this.
-

[Paul updates issue]
-

Next steps
-

Discussion on how to get more feedback on issues
-

Sorting by thumbs up is somewhat useful, but there are too many
different emoji (heart, smiley face, etc.)
-

Ask CG members to thumbs up the issues that are important to them.
-

Headphone detection
<https://github.com/WebAudio/web-audio-api-v2/issues/89>
-

Hongchan: self-assigned
-

Moving to priority-1
-

Channel layout detection for output device
<https://github.com/WebAudio/web-audio-api-v2/issues/87>
-

Hongchan: Should be handled by media output devices
-

Hongchan: Not sure if layout detection is possible. Also exposes
more fingerprinting.
-

Paul: I can tell you but it will take a long time. Stereo output
could actually be channels 7 and 8 on a fancy audio card.
-

[Paul updates issue]
-

Hongchan: Ask Phillipe how this works in Wyse.
-

[See also discussion above about output device selection
<https://docs.google.com/document/d/1l8yniQXgOUTNwEO7-I4YZBiW11-NrXz3L4zdNr156Io/edit#bookmark=id.caizb7u35cg5>
.]
-

Meetings adjourned for TPAC 2020.
-

We’ll resume our regularly scheduled meetings next week at the usual
time and place.
-

Thanks to everyone who participated!

Received on Friday, 23 October 2020 20:17:18 UTC