Fwd: Dynamic Lifetime and AudioWorker

---------- Forwarded message ----------
From: Chris Wilson <cwilso@google.com>
Date: Wed, Feb 4, 2015 at 12:24 PM
Subject: Re: Dynamic Lifetime and AudioWorker
To: Steven Yi <stevenyi@gmail.com>


On Tue, Feb 3, 2015 at 9:10 PM, Steven Yi <stevenyi@gmail.com> wrote:
>
> >> 1. Will it be possible for an AudioWorker to signal a release,
> >> similarly to how the above AudioBufferSourceNode can signal turning
> >> off? (An imagined use case is below.)
> >
> >
> > Well, sure.  Set its onaudioprocess to null; if there are no references to
> > it anywhere, it will get garbage collected.
>
> I have not seen anything in the spec (I'm using the one at
> http://webaudio.github.io/web-audio-api/, dated 06 Jan 2015) regarding
> setting onaudioprocess to null and what that means.  Is this specified
> in a more recent document?


No; it's pragmatic.  If you've released all references to an object,
and that object is no longer doing any processing, and can NEVER do
any processing, it can be collected.

>
> Does that mean that from within an
> onaudioprocess, one can do:
>
> onaudioprocess = function(e) {
>    ....
>    if(done) {
>      this.onaudioprocess = null;
>    }
> }
>
> ?


Yeah, not positive the scoping works properly without more context,
but essentially yes.  As long as you don't have any other object
references anywhere.

>
> Also, how does this relate to the terminate() function that's exposed
> in the AudioWorkerNode?


You could use that, too.  terminate() is the direct version of doing
that (forgot we picked that up from Web Workers).

>
> BTW: I think in terms of release, I should have said something like
> "turning off, resulting in an auto-disconnect".  It seems then that if
> one sets onaudioprocess to null, one gets the same auto-disconnection
> as is described by ABSN when it's done reading it's buffer?


Yes.  (Again, it's not just setting onaudioprocess to null; it's that
PLUS not having any object references to the worker, so you could
never revive it.)

>
> Perhaps a better question: it seems that when ABSN is done with
> playback, it no longer has a "playing" reference on itself, and that
> is part of what trigger the auto-disconnection.  If correct, should
> AudioWorkers have the same ability to have a "playing" reference?  Or
> if that "playing" reference aspect is tied to the presence or absence
> of an onaudioprocess, could that be added to the specification?


They essentially do, and yes, it's having an event handler.  Added an
issue to capture needing to detail that
(https://github.com/WebAudio/web-audio-api/issues/475).

> >> 2. Does this release signal correspond to the "onended" event handler,
> >> discussed in 2.9.1?
> >
> >
> > Not at all.  "onended" does not correlate to the release of an
> > AudioBufferSourceNode - in fact, it can't, because if you're firing the
> > event you have a reference to the node object, and that will prevent it from
> > releasing.
> >
> > Also, what triggers an ABSN onended is reaching the end of the buffer in
> > playback; what should trigger the onended of a worker?  If your answer is
> > "that's up to the work implementation", then sure - but that just means you
> > want to post an ended event from inside your audio worker back to the main
> > thread, which you can absolutely do.  (And set onaudioprocess to null, if
> > you like.)  In short - you can do this already, you don't need extra code in
> > AudioWorker to be allowed to do so.
>
> Again sorry, I should have said something like auto-disconnect here
> rather than release as in GC freeing.  My question in 2. was more
> concerned with the ability of an AudioWorker to auto-disconnect from
> the graph than any memory related issues.  The use of onended as just
> a notification for the user and having nothing to do with internal
> releasing or auto-disconnect is clear to me now from your reply,
> thanks.


It's pretty much the same.  AudioWorker can't auto-disconnect from the
graph until the system can be certain that it will never again be
needed (or perform processing).  That means 1) no object references to
it - or an onaudioprocess could be added again, and the connections
should be valid), and 2) onaudioprocess does not have an event
handler, or it would be processing.

>
> >> 5. Regarding Example 4.2, Figure 7, if the Compressor did not have any
> >> the Stream Source and Gain 1 nodes, what prevents the Compressor node
> >> from also being released when the AudioBufferSourceNode is released?
> >> >From reading the spec, it seems like it would get killed off once the
> >> last Node that is attached is detached.
> >
> >
> > If the streaming source (and gain node) were not present, it WOULD be
> > released.  The streaming source and gain node are what keeps it alive.
> > (again, this presumes you've released any JS references to the
> > DynamicCompressor; otherwise it would be maintained in case you connected
> > something else to it.
>
> Ah okay, so as long as there is JS ref to the DynamicCompressor, it
> will not auto-disconnect.  This brings up another question:
> auto-disconnection seems to be dependent on if a node is release-able
> in the GC sense.  If that is correct, does that mean if one keeps a
> ref, that nodes will not auto-disconnect? Say for that example, if the
> user did have a JS ref to the to the One-Shot sound, once that buffer
> is complete, does that whole chain within the dotted lines remain
> connected to the graph and processing of the nodes continue to occur
> (until that last JS ref is released)? (I guess in that scenario it
> would be up to the user to use the onended event to explicitly call
> disconnect()?)


It's tough to answer that question.  If the user had a JS ref to the
one-shot sound in that graph, then I'd say that yes, the graph would
remain connected.  However, processing of nodes would NOT continue to
occur, because all the nodes in the rest of the graph (lp filter,
panner, gain) don't have a time component - their output is silent if
their input is silent - so as an optimization, they could skip
processing.  Similarly, and more importantly, if you don't have any JS
refs on those objects, the whole graph is dead, since the
one-shot-sound (an AudioBufferSourceNode) can't be re-used (thus,
can't ever generate sound), and you have no way to access the interim
nodes, so they'll never get an input, and thus never have any output.

A more insightful example would be if you maintained a JS reference to
the LowPass filter node, but NOT the one-shot sound node.  Once the
one-shot sound finished, it would auto-disconnect and be released;
however, the filter, panner and gain would remain connected.  (Since
you have a JS ref to the filter, you might connect a NEW node to it,
and then it would be processing audio again).

> >> Use Case for AudioWorker: A user initiates a "note" via a MIDI
> >> keyboard.  An AudioWorker that implements a combined envelope
> >> generator + VCA is used to process the output of a set of
> >> OscillatorNodes, and these are all added to the main audio graph. When
> >> the user depresses the key, a message is posted to the AudioWorker to
> >> go into a release stage.  At the end of the release stage, the
> >> AudioWorker could fire an end event, cleaning up the set of nodes.
> >
> > You can do this today.  You don't need an end "event" - you just destroy the
> > node.
>
> Okay, just to clarify, when you say "destroy the node", does that mean
> to initiate that by the AudioWork itself setting onaudioprocess =
> null, as mentioned earlier? (Assuming that releases a "playing"
> reference on itself).


Or calling terminate(), yes.  Again, as long as JS refs aren't being held on it.

> >> 2. If 1 above is a correct interpretation, will it be possible for an
> >> AudioWorker to specify it's own tail-time?
> >
> > As the audio worker manages its own lifetime, yes it can.
>
> Okay, I think this is getting a little hairy. I'm not sure then how
> onaudioprocess and "playing" and "tail-time" references here could
> work.  Let's say we have an AudioWorker that implements a reverb
> that's part of "note" chain of nodes. For this let's assume there's an
> ABSN at the head of the list of nodes. There are no JS refs once the
> node is connected into the graph.  For the user-defined AudioWorker to
> have the same lifecycle as a ConvolverNode, it would need to be able
> to explicitly set a tail-time to get a "tail-time" reference, but also
> not have any kind of "playing" reference, so that the node can
> auto-disconnect once the tail-time is done.
>
> So how would this work? Is there a plan to expose a tailTime
> read/write property in AudioWorkerGlobalContext, so that the node can
> set that?


No, because the onaudioprocess can decide when it's done processing
(and when to destroy itself, effectively deciding what its tail time
is).  It might be interesting for the end user, but it's informative,
not normative.

> > No, that's as expected - you probably in this case have a normal JS
> > reference.  If you do NOT have a normal JS reference, then the Panner could
> > (and should) be garbage collected, as there is no way it is ever going to
> > generate sound in the future (how would you connect anything to it with no
> > JS reference?).
>
> Ah that makes sense. I think I didn't have the rules of the different
> references and the relationship to auto-disconnect clear in my head
> when I made this comment. By that logic though, let's say a cycle of
> nodes is created and is connected to the AudioDestination, but all of
> the JS refs are cleared.  Does that mean it's possible to have a group
> of nodes where there is no way for the user to disconnect and clean
> them up without killing the audiocontext?


Yes.

Received on Wednesday, 4 February 2015 18:08:25 UTC