Event dispatch relative to AudioContext time

Hi Steven,

Many points here worth responding to, but I will just reference your #5 since it was also raised by some other people at the WAC and I think it is an important issue for both realtime and offline audio rendering.

Please see the newly filed https://github.com/WebAudio/web-audio-api/issues/473

…Joe



On Jan 30, 2015, at 7:57 PM, Steven Yi <stevenyi@gmail.com> wrote:

> Hello All,
> 
> First, it was a great pleasure to be at the Web Audio conference. I
> enjoyed the sessions and gigs and getting to the meet the other
> members of community that I did.  Cheers to IRCAM and Mozilla for the
> lovely conference!
> 
> That said, I have some comments and questions about the Web Audio API
> and specification. (Note: these comments are in reference to the 06
> January 2015 draft, found at
> http://webaudio.github.io/web-audio-api/.)
> 
> #1 - The specification is not clear to me when a node become live. I
> assume it is when a node is connected to the active part of the audio
> graph that is "live" and processing. Since node creation and graph
> assembly is done in the JS Main thread, it seems that the following
> from "3.3 Example: Mixer with Send Busses", it possible that nodes
> might get attached across buffers in the audio thread:
> 
>   compressor = context.createDynamicsCompressor();
> 
>    // Send1 effect
>    reverb = context.createConvolver();
>    // Convolver impulse response may be set here or later
> 
>    // Send2 effect
>    delay = context.createDelay();
> 
>    // Connect final compressor to final destination
>    compressor.connect(context.destination);
> 
>    // Connect sends 1 & 2 through effects to main mixer
>    s1 = context.createGain();
>    reverb.connect(s1);
>    s1.connect(compressor);
> 
>    s2 = context.createGain();
>    delay.connect(s2);
>    s2.connect(compressor);
> 
>  For example, could it be the case that "s1.connect(compresor)" above
> happens just before buffer n starts to generate, and
> "s2.connect(compressor)" happens such that it starts in when buffer n
> + 1 is generating?
> 
> If this is the case, would connecting the compressor to the
> context.destination at the end of the example, rather than the
> beginning, guarantee that the graph of nodes connected to the
> compressor are started at the same time?  If so, then maybe this
> aspect of node graph creation could be clarified and the example in
> 3.3 updated so that the sub-graph of nodes is clearly formed before
> attaching to the active audio-graph.
> 
> #2 - Following from #1, what would happen if one is dynamically
> altering a graph to remove an intermediary node?  For example, lets
> say one has a graph like:
> 
>   gain = contxt.createGainNode();
>   compressor = context.createDynamicsCompressor();
>   reverb = context.createConvolver();
>   gain.connect(reverb);
>   reverb.connect(compressor);
>   compressor.connect(context.destination);
> 
> and later the user decides to remove the reverb with something like:
> 
>   reverb.disconnect();
>   // gain.disconnect();
>   gain.connect(compressor);
> 
> (Assuming the above uses a gain node as a stable node for other nodes
> to attach to.) My question is: when does connect and disconnect
> happen?  Does it happen at block boundaries?  I assume it must or a
> graph can get in a bad state if the graph changes while a block is
> being processed.
> 
> Also, without the gain.disconnect(), will there be a hidden reference
> to the reverb from gain? (I guess a "connection" reference according
> to 2.3.3). If so, this seems like it could be a source of a memory
> leak (assuming that the above object references to reverb are all
> cleared from the JS main thread side).
> 
> #3 -  In "2.3.2 Methods", for an AudioNode to connect to another audio
> node, it is not clear whether fan-out/fan-in is supported.  The
> documentation for connecting to AudioParams explicitly states that
> this is supported.  Should the first connect() method documentation be
> clarified for this when connecting to nodes?
> 
> #4 - Also in regards to 2.3.2, the API of disconnect() seems odd as it
> does not mirror connect(). connect() is given an argument of what node
> or audioParam to connect to.  disconnect() however does not have a
> target argument. It's not clear then what this disconnects from. For
> example, if I connect a node to two different nodes and also to
> another node's parameter, then call disconnect, what happens?  As it
> is now, it doesn't seem possible then to create a GUI editor where one
> could connect the output of a node to multiple other nodes/params,
> then click and disconnect a single connection.
> 
> #5 - In the music systems I've seen, event processing is done within
> the audio-thread.  This generally happens for each buffer, something
> like:
> 
> 1. Process incoming messages
> 2. Process a priority queue of pending events
> 3. Handle audio input
> 4. Run processing graph for one block
> 5. Handle audio output
> 
> I'm familiar with this from Csound and SuperCollider's engines, as
> well as the design in my own software synthesizer Pink. (Chuck's
> design follow the same basic pattern above, but on a sample-by-sample
> basis.)
> 
> As it is today, the Web Audio API does not have any kind of reified
> event object.  One can schedule some things like automations via
> param's setXXXatTime() methods and have that run within the time of
> the audio engine, but there is nothing built-in for events in the Web
> Audio API.
> 
> Now, I have no issues with the Web Audio API not having a concrete
> event system, and think it should not have one, as people have
> different notions and needs out of what is encoded in an event.
> However, I think that there should be a way to create one's own event
> system, one that is clocked to the same audio system clock (i.e. run
> within the audio thread).
> 
> I was a bit concerned when at the conference there was mention of "A
> Tale of Two Clocks".  The design of trying to reference two clocks can
> not, by definition, allow for a queue of events to be processed
> synchronously with audio. If one formalizes events processing
> functions and audio processing functions as functions of time, by
> having two clocks you get two different variables, ta and tb, which
> are not equivalent unless the clocks are proven to advance at the same
> exact rate (i.e. ta0 == tb0, ta1 == tb1, ... tan == tbn).  However,
> the JS Main thread and audio thread are not run at the same rate, so
> we can at best implement some kind of approximation, but it can not be
> a formally correct solution.
> 
> Event processing in a thread other than the audio thread has problems.
> One mentioned at the conference was what to do with offline rendering,
> where the clock of an audio engine runs faster than realtime, and may
> advance faster or slower in terms of wall-clock time while rendering,
> depending on how heavy the processing needs of the graph is.  Second,
> I seemed to remember hearing a problem during one of the concerts when
> I turned off my phone's screen and I continued to hear audio but all
> events stopped, then a number of events fired all at once when I
> turned my screen back on. The piece used an event scheduling system
> that ran in the JS Main thread. I assume this situation is similar to
> what could happen with backgrounded tabs, but I'm not quite sure about
> all this. Either way, I think there are real problems here that need
> to be addressed.
> 
> This also leads to a bigger question: with Web Audio, if I run the
> same project twice that uses an event system to reify graph
> modifications in time (as events in audio engines are mostly used for,
> i.e. alloc this graph of nodes and add to the live audio graph), will
> I get the same result?  Assuming to use only referentially transparent
> nodes (i.e. no random calculations), I believe the only way to
> guarantee this is if the event system is processed as part of the
> audio thread.
> 
> Now, what can a user do with Web Audio to create their own Event
> system that is in sync with the audio thread?  Currently, there is the
> ScriptProcessorNode.  Of course, the design of ScriptProcessorNode is
> deeply flawed for all the reasons discussed at the conference
> (Security, Inefficient due to context switching, potential for
> breakups, etc.).  However, what it does do is allow for one to process
> events in sync with the audio thread, allowing to build formally
> correct audio systems where one processes event time according to the
> same time as is used by the audio nodes. Additionally, according to
> those events, one can dynamically modify the graph (i.e. add new
> instances of a sub-graph of nodes to the live graph, representing a
> "note"), via reference to other nodes and the audio context. So while
> flawed in terms of performance and security, it does allow one to
> build correct systems that generate consistent output.
> 
> My concern is that there was discussion of not only deprecating
> ScriptProcessorNode, but removing it altogether.  I would have no
> problems with this, except that from reading the current specification
> for AudioWorker, I do not see how it would be possible to create an
> event system with it.  While one can pass messages to and from an
> AudioWorker, one has no access to the AudioContext. In that regards,
> one can not say, within an AudioWorker, create new nodes and attach to
> the context.destination. I am not very familiar with transferables and
> what can be passed between the AudioWork and the JS Main thread via
> postMessage, but I assume AudioNodes can not be made transferable.
> 
> At this point, I'm questioning what can be done. It seems
> AudioWorker's design is not meant for event processing (fair enough),
> and ScriptProcessor can only do this by accident and not design. Is
> there any solution to this problem with the Web Audio API moving
> forward?  For example, would this group be willing to consider
> extending the API for non-audio nodes?  (Processing nodes?). If
> processing nodes could be added that has a larger context than what is
> proposed for AudioWorkGlobalContext--say, has access to the
> AudioContext, and can modify the audio node graph dynamically--I could
> see it as a solution to allow building higher level constructs like an
> event system.
> 
> #6 - For the AudioWorker specification, I think it would be useful to
> have clarification on when postMessage is processed.  In 2.11.1.2, it
> has a link to "the algorithm defined by the Worker Specification".
> That in turn mentions:
> 
> "The postMessage() method on DedicatedWorkerGlobalScope objects must
> act as if, when invoked, it immediately invoked the method of the same
> name on the port, with the same arguments, and returned the same
> return value."
> 
> If it meant to be processed immediately, then this can cause problems
> if the AudioWorker is already part of a live graph and values mutate
> while an audio worker is processing a block. I think it would be good
> to have clarification on this, perhaps with a recommendation that in
> onaudioprocess functions, one should make a local copy of a value of a
> mutable value and use that for the duration of onaudioprocess to get a
> consistent result for the block.
> 
> #7 - Related to #6, I noticed in "2.11.3.1 A Bitcrusher Node", the
> example uses a phaser variable that is scoped to the AudioWorker.  I
> assume this would then be on the heap. This is perhaps more of general
> JS question, but I normally see in block-based audio programming that
> for a process() function, one generally copies any state variables of
> a node/ugen/etc. to local variables, runs the audio for-loop with
> local variable, then saves the state for the next run.  This is done
> for performance (better locality, stack vs. heap access, better
> compiler optimizations, etc.). I don't know much about JavaScript
> implementations; can anyone comment if these kinds of optimizations
> are effective in JS?  If so, the example might benefit from rewriting
> and give some guidance. (i.e. phase and lastDataValue are copied to a
> local var before the for-loop, and saved again after the for-loop, in
> onaudioprocess).
> 
> Thanks!
> steven
> 
> 

.            .       .    .  . ...Joe

Joe Berkovitz
President

Noteflight LLC
Boston, Mass.
phone: +1 978 314 6271
www.noteflight.com
"Your music, everywhere"