W3C home > Mailing lists > Public > public-audio@w3.org > July to September 2012

Re: Web Audio API sequencer capabilities

From: rl baxter <baxrob@gmail.com>
Date: Wed, 22 Aug 2012 00:04:22 -0700
Message-ID: <CACo9a0coq_La4Hi6kvx15vGO6bV-PT0Jyxd0tJfTpL5dJ6CfAA@mail.gmail.com>
To: Jussi Kalliokoski <jussi.kalliokoski@gmail.com>
Cc: public-audio@w3.org
Hello all,

This reminded me:

1. Speaking of sequence event timing, the idea of a callbackAtTime
method has been raised, though not much discussed AFAIK.  At least,
maybe this webkit bug report should be ported to the w3 tracker:
https://bugs.webkit.org/show_bug.cgi?id=70061 .
  This idea speaks volumes to me because there is currently no way to
schedule arbitrary events in sync with the audio clock (the closest
thing is firing an event in the jsNode.onaudioprocess loop, but that
can only be accurate to within the time-grain of its buffer size).

2. Re: "breaking changes".  They will only become more breaking, no?
It'd be least harmful to incorporate any glaringly necessary ones as
soon as possible.  For example, it seems clear that
JavascriptAudioNode calls for severe re-thinking.

2,
Roby


On Tue, Aug 21, 2012 at 11:30 AM, Jussi Kalliokoski
<jussi.kalliokoski@gmail.com> wrote:
> I forgot a few things I had in mind.
>
> Usually in a DAW, if you press pause, effects such as reverbs etc. will
> still play their tails. This is somewhat the default behavior with the Web
> Audio API, and when plugins would receive a pause event they could just call
> cancelScheduledValues on AudioParams. However, if you press stop (or panic
> in some cases), the audio gets cut off immediately, and all the active notes
> in virtual instruments get killed (the latter is true with pause as well).
> How would one replicate this behavior with the Web Audio API? One could just
> drop the connection to the audio destination, but AFAICT then when the
> playback starts again, the tails would get played on top of that, which
> would most likely be undesirable.
>
> My suggestion for this is that the native nodes would have a reset() method
> (not attached to the name) that would make them forget clear their internal
> buffers, i.e. cancel all the echo/reverb tails and reset the internal state
> of filters and an oscillator's phase.
>
> Talking about Oscillators brings me to another subject. As the current
> source nodes that are useful in synthesis (oscillator and buffer) are
> designed to be simple shot only, the idea of the plugins exposing desired
> AudioParams doesn't work anymore. Now this is OK, because the idea is
> actually flawed from another perspective as well, as it may be that the host
> is scheduling value changes to the params simultaneously with the plugin,
> which would probably lead to some very unexpected results (from the POV of
> the DAW user). This can be worked around by exposing a gain node that pipes
> values to the AudioParam(s) instead. But now the host loses the ability to
> efficiently schedule values / envelopes to the parameters.
>
> Suggestion: We separate the envelope / scheduling behavior from the
> AudioParams to a new native source node, something like an EnvelopeNode.
>
> Other benefits:
>  * You could have multiple envelopes for a single AudioParam.
>  * You could have a single envelope for multiple AudioParams.
>  * AudioParam would be a whole lot simpler an interface.
>  * The need for a node that produces a certain offset would be solved.
>  * Probably a lot more.
>
> The only downside I can think of is that it's a breaking change. It's a big
> one, but then again, this is a working draft with only one working
> independent implementation anyway.
>
> Cheers,
> Jussi
>
>
> On Tue, Aug 21, 2012 at 8:58 PM, Jussi Kalliokoski
> <jussi.kalliokoski@gmail.com> wrote:
>>
>> Hello group,
>>
>> I've been thinking about how to use the Web Audio API to write a
>> full-fledged DAW with sequencing capabilities (e.g. MIDI), and I thought I'd
>> share some thoughts and questions with you.
>>
>> Currently, it's pretty straight-forward to use the Web Audio API to
>> schedule events in real time, which means it would play quite well together
>> with other real time APIs, such as the Web MIDI API. For example, you can
>> just schedule an audiobuffer to play whenever a noteon event is received
>> from a MIDI source.
>>
>> However, here's something of a simple idea of how to build a DAW with a
>> plugin architecture using the Web Audio API:
>>
>>  * You have tracks, which may contain audio and sequencing data (e.g.
>> MIDI, OSC and/or user-defined envelopes). All of these inputs can be either
>> being recorded from an external source, or be static pieces.
>>
>>  * You have an effects list for each track, effects being available to
>> pick from plugins.
>>
>>  * You have plugins. The plugins are given references to two gain nodes,
>> one for input and one for output, as well as a reference to the
>> AudioContext. In response, they will give AudioParam references back to the
>> host, as well as some information of what the AudioParams stand for, min/max
>> values and so on. The plugin will set up a sub-graph between the given gain
>> nodes.
>>
>> This would be a very basic setup, but with the current API design there
>> are some hard problems to solve here. The audio is relatively easy,
>> regardless of whether it's coming from an external source or not. It's just
>> a source node of some sort. The sequencing part is where stuff gets tricky.
>>
>> In the plugin models I've used, the sequencing data is paired with the
>> audio data in processing events, i.e. you're told to fill some buffers,
>> given a few k-rate params, a few a-rate params and some sequencing events as
>> well as the input audio data. This makes it very simple to synchronize the
>> sequencing events with the audio. But with the Web Audio API, the only place
>> where you get a processing event like this is the JS node, and even there
>> you currently only get the input audio.
>>
>> What would be the proposed solution for handling this case? And please, no
>> setTimeout(). A system is as weak as its weakest link and building a
>> DAW/Sequencer that relies on setTimeout is going to be utterly unreliable,
>> which a DAW can't afford to be.
>>
>> Cheers,
>> Jussi
>
>
Received on Wednesday, 22 August 2012 07:04:52 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 22 August 2012 07:04:55 GMT