Re: [w3ctag/design-reviews] Web Audio API: RenderCapacity API (Issue #843)

Audio systems, when rendering an audio stream, typically work with a synchronous audio callback (called in the spec a `system level audio callback`), that is called in an isochronous fashion, by the system, on a real-time thread, with a buffer of `n` frames, that the program must fill entirely, and the return as soon as possible.

This callback, called continuously during the lifetime of the audio stream, returns to the system with audio samples, and then hands it off to the rest of the OS. This audio might be post-processed and is usually output on an audio output device, such as headphones or speakers.

Let `frames[i]` be the number of frames that has to be rendered by the callback on the `i`-th iteration (a few hundreds to a couple of thousands is typical in this situation)
Let `sr` is the sample-rate at which the audio system runs (44100Hz or 48000Hz are typical values)
 Let `r[i]` the time, in seconds, it took to render `n` frames this time. This is in other words, the execution time of the callback

`frames[i] / sr` is a number of audio frames, divided by the sample-rate, so it's a duration in seconds. It's the duration a buffer of `frames[i]` samples takes to be played out.

The load for this render quantum is:

> load[i] = r[i] / (frames[i] / sr)

In a nominal scenario, the load is below 1.0: it took less time to render the audio than it takes to play it out. In an overload scenario (called under-run in the audio programming jargon), the load can be greater than 1.0. At this point, it is expected that the user will hear audio dropouts. This provokes discontinuities in the audio output and is very noticeable.

Because the time it takes to render the audio is usually directly controllable by authors (for example, by deciding to reduce the quality of some parts of the audio processing graph, that are less essential than others for the application), authors would like to be able to observe this load.

A real-life example that could benefit from this new API would be the excellent https://learningsynths.ableton.com/. If you open the menu by clicking the icon on the top left (on desktop), and scroll down this panel, you see that the render quality is controllable.

Similarly, it's not uncommon for digital audio workstations or other professional audio software to display a load indicator in their user interface, to warn the user that there's too much processing for the system in its current configuration.

In the Web Audio API spec, this is defined in the section [Rendering an audio graph](https://webaudio.github.io/web-audio-api/#rendering-loop).


-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3ctag/design-reviews/issues/843#issuecomment-1663473030
You are receiving this because you are subscribed to this thread.

Message ID: <w3ctag/design-reviews/issues/843/1663473030@github.com>