Re: Status v2

HI Paul,

     Thanks for taking the time for these thoughtful responses to 
Andre's questions! My insight into the thinking of the group just went 
way up.

Best,
          - lonce--

Lonce Wyse, Associate Professor
Communications and New Media Department; Director, SSI Arts and 
Creativity Lab
Editorial Boards: Organized Sound 
<https://www.cambridge.org/core/journals/organised-sound>; Computer 
Music Journal <https://www.mitpressjournals.org/cmj>; International 
Journal of Performance Arts & Digital Media (IJPADM) 
<http://explore.tandfonline.com/cfp/ah/rpdm-journal-performance-arts-digital-media>
National University of Singapore
Address: AS6, level 3, 11 Computing Dr., Singapore 117416 National 
University of Singapore
lonce.org/home <http://lonce.org/home>

On 18/4/20 1:38 am, Paul Adenot wrote:
> André,
>
> On Thu, Apr 16, 2020 at 9:07 PM André Michelle 
> <andre.michelle@audiotool.com <mailto:andre.michelle@audiotool.com>> 
> wrote:
>
>
>     is there some kind of summary for the most important key features?
>     There are a lot of issues on github that deal with details of
>     existing audio-nodes.
>
>
> There are some improvements, because people want to use audio nodes, 
> and would like them extended. But I agree that a good triage of the 
> backlog would be in order, it's a mess at the minute.
>
>     I think for many developers that are building complete DAWs with
>     their own infrastructure, improving audio-nodes is not that
>     interesting. We are looking for the most stable solution to run
>     our own dsp-code in a different prioritised thread without any
>     performance crucial dependencies like the main gc,  event loop or
>     graphic updates (while I understand that the communication between
>     those threads will introduce message-latency when the browser is
>     busy, but that is neglectable).More important is getting rid of
>     glitches (pops, clicks), that the main thread introduces.
>
>
> No amount of specification will fix that, this is clearly an 
> implementation detail (apart from things like 
> https://github.com/WebAudio/web-audio-api/issues/1933, and that's 
> stretching it). Communication between threads doesn't have to add 
> latency, we have SharedArrayBuffer now, see below.
>
>     We are already pooling objects whenever they have a short lifetime
>
>
> (unrelated, but this is an anti-pattern in our day and age of 
> generational GC in modern JS engines, if you're using a GC-ed langage, 
> https://hacks.mozilla.org/2014/09/generational-garbage-collection-in-firefox/ 
> explains why, but this is in no way limited to Firefox)
>
>     , but the overall performance seems to be worse than in Flash
>     (where everything ran in the main thread). But that is of course
>     a subjective observation, since we cannot compare both properly.
>     However many users ask for the Flashversion of audiotool, because
>     it ran better for them. Especially on Chromebooks. Many schools in
>     the US are using audiotool on Chromebooks.
>
>     How can I contribute?
>     My guess is that fixing those issues is rather about the internal
>     infrastructure of the browser than specifications. In theory the
>     audio-worklet is the way to go.
>
>
> It is. We've investigated a simpler API (called Audio Device Client), 
> but after some time implementing AudioWorklet, it became apparent that 
> we could not get any more performances with a different API design. If 
> this conclusion is proven to have been wrong, there is room in the Web 
> Audio API charter to investigate another solution than AudioWorklet.
>
> The internals of browsers don't really have any influence on 
> specification work (sometimes it's the case, but in general it's 
> explicitly against the design rules of the web platform).
>
>
>     The only case I can make is to have an adjustable
>     audio-ring-buffer (relaxing the 128 frame block-size) to better
>     handle short performance peaks and make it completely independent
>     from the main-thread. I understand, it is easier said than done.
>     Does the v2 specifications include that?
>
>
> It does, but 128 frame block size is not an IO vector size (as it's 
> often called). It's a block processing size. Browsers can (and do) 
> choose a different IO vector size, and you can change it yourself for 
> what is best for your application using the `latencyHint` parameter, 
> when creating and AudioContext: 
> https://webaudio.github.io/web-audio-api/#dom-audiocontextoptions-latencyhint.
>
> Changing the buffer size is however 
> https://github.com/WebAudio/web-audio-api-v2/issues/13, because it has 
> other performance implications as explained in the issue.
>
> The AudioWorklet is already completely independent of the main thread 
> (or any other thread or process for that matter) and can run on a 
> real-time thread, we've been very careful to specify it this way (part 
> of the reason why it took so long).
>
>     Are you trying to merge v2 into v1?
>
>
> That's the idea I think (it has not been decided), however it's 
> unclear if it is something that we'll succeed in doing. It's quite 
> challenging in terms of backwards compat.
>
>     That would explain the issues for existing audio-nodes. I guess
>     for most advanced audio programmers a parallel v2 with a very
>     simple api (config and push callback like it was in pepper) would
>     solved many problems we are facing right now.
>
>
> This would provide no improved performances, and you can already do it 
> today. Careful and extensive measurements have shown that the overhead 
> of the AudioWorklet can be made minimal (and is, in Firefox), compared 
> to C code running outside of a browser. WASM speed is enough for a lot 
> of uses as well (but it would be lying to say it's as fast as native 
> code, today, we see a 1.5x to 2x slowdown on the type of workloads we 
> care about here).
>
> Having a config and push callback would be a regression in terms of 
> latency and reliability, but if you want to do it (there are good 
> reasons), you can do it with AudioWorklet. Because I know this is a 
> common request, I've written a library that does the heavy lifting for 
> this, in a way that is the most performant possible: 
> https://github.com/padenot/ringbuf.js.This is very liberally licensed 
> intentionally, but I'm happy to hear about any feedback (positive or 
> negative). This would be by definition a priority inversion however, 
> so care must be taken to buffer enough to avoid underruns.
>
>
>     This may sound blunt, but let the web-developers build their own
>     audio-frameworks. In no time we will have all the currently
>     existing v1 audio-nodes covered and way more in the future. The
>     actual issue is not making better nodes. Developers will always
>     ask for more. But giving them a universal access point to a
>     prioritised, configurable thread will certainly create more
>     impact, very similar to WebGL (where many frameworks were built
>     with amazing features for different tasks).
>
>
> This is what developers have been doing for some time with 
> AudioWorklet, and you can do it today (other people/company/projects 
> have done it with great success, in particular compiling commercial 
> programs into WASM and running them on the web).
>
> What you're describing is exactly what AudioWorklet is: a prioritized 
> thread, with a configurable audio output callback (in terms of 
> latency, device, channel count, and at some stage in v2 block 
> processing size), but on top of that it is composable with other 
> nodes, so if you want to stream any audio from the web it's a couple 
> lines of code, and if you want a multi-threaded convolver it's another 
> two lines of code. Opposing AudioWorklet/custom DSP and regular audio 
> nodes is not useful. Some use-cases are better implemented in one, 
> others using the other technique, and quite a lot are using both at 
> the same time.
>
> Hope this helps,
> Paul.