Re: Audio Workers - please review

We’re mostly in agreement here, but I have a strong feeling that you’re misunderstanding something that I’m saying, or vice versa. Probably I’m not saying it (or understanding you) clearly.

I just don’t see how parallelization can have any observable (audible) effect on emitted audio whatsoever, as long as the total processing cost of the entire graph doesn’t cause glitching or underrun at the output. After all, in any automatically parallelized system, the implementation would be responsible for ensuring that a given range of sample frames proceeded through parallel paths (even if the paths have dissimilar processing costs) and then, at some node acting as a join point, would be reintegrated with sample-accurate matching. The fact that multiple threads are doing the work should have no computational effect at all.

When I say “insert latency”, I only mean, “a processing delay in the graph (e.g. inter-thread handoff of a buffer) that has no computational effect: it does not change either the number or the value of any sample frames in the path”. Some people on the group have used the term “latency” in this way at times.

But I have a feeling that when you say, “insert latency”, you mean “an actual delay that adds some number of sample frames into a processing path”.  Or maybe you mean, “a processing cost which requires additional buffering around the graph”. In which case, I understand your point of view.

Does this clear anything up? Because I’m not proposing that we accommodate any future changes that could have any effect on the behavior of the graph, or its computational result, or the ability to handle a previously reasonable buffer size / total latency combination. That would be bad indeed.

Perhaps we need a lexicon to help us (or, at least, help me).

.            .       .    .  . ...Joe

Joe Berkovitz
President

Noteflight LLC
Boston, Mass.
phone: +1 978 314 6271
www.noteflight.com
"Your music, everywhere"


On Sep 12, 2014, at 12:46 PM, Chris Wilson <cwilso@google.com> wrote:

> On Fri, Sep 12, 2014 at 5:04 PM, Joseph Berkovitz <joe@noteflight.com> wrote:
> I suggest that we approach this issue not from the standpoint of what we should do now in the API, or even soon — rather, the question is, should we adopt a stance that rules out what may be a useful approach in the future, whose feasibility might be open to doubt today and later become very clear. I propose that the API avoid a stance that implicit parallelization is to be avoided for all time, and avoid being skewed in favor of explicit parallelization in a permanent fashion.
> 
> If you are saying the Web Audio system should reserve the right to arbitrarily (to the developer) insert latency at various points in the node graph in order to parallelize, I feel that we are passing the time that needs to be defined.  My vocoder demo, for example, would get pretty messed up if some of the graph got moved into another thread with added latency, and some didn't.
> 
> If you are saying we want to reserve the possibility to parallelize intuititively when we can work ahead to avoid adding latency into parts of the graph, I don't have a problem with that, I'm just highly skeptical it's worthwhile.  (The vocoder, for example, probably couldn't be broken apart.)
> 
> As previously, I think parallelization should be possible for developers to implement if they wish (inserting latency intelligently).
>  
> We need to leave room for flexibility and avoid reaching premature conclusions. Mostly this just means avoiding global scopes that rule out parallelism, and avoiding overly specific definitions of behavior in the spec.
> 
> I agree that we should not have global scopes (i.e. nodes sharing things they don't need to share); I disagree about avoiding specific definitions of behavior in the spec.  The spec needs to be much MORE specific than it is today.
>  
> I know for a fact that some native DAWs do arbitrary parallelization as a matter of course in isolated linear effect chains, and that it does not incur an unacceptable latency cost.
> 
> But that's not arbitrary (nor "automatic" in the sense that I mean).  They're inserting parallelization at specific points (isolated effect chains), and the developer is choosing to do it.  That's exactly the kind of parallelization I want to make sure we have enabled.  What I'm skeptical of would be more akin to "Core Audio now arbitrarily may put 20ms latency into random nodes."
>  
> I think we all know a couple of pro audio app builders, but perhaps not the same ones :-)  So automatic parallelization is done already outside the web, and it’s apparently considered quite a good idea in at least some contexts. Don’t UAs already parallelize lots of activity on the user’s behalf without exposing it?
> 
> Some, when it can be parallelized without side effects; but largely not.  That's why that main thread is so congested.  :)
>  
> Also, the cost of graph analysis will drop over time. I don’t see offhand why the latency it adds is necessarily of a showstopper variety. You said, "But I think inserting latency at ANY point in the graph connections, NOT at the explicit request of the developer, is a bad idea.” However, if inserting latency at some point in a subgraph decreases overall latency in the graph as a whole… I don’t know, that seems like a pretty good thing to me, in theory.
> 
> But that's not what we're doing.  There isn't "overall latency in the graph" - the graph's latency is zero (the latency in Web Audio is at the input and the output).  The tradeoff here would be CPU bandwidth (balancing the work across multiple cores - aka lower likelihood of glitching) for latency. 
>  
> But perhaps we don’t have to prove it either way right now :-)
> 
> If you plan on changing the API at some point so it inserts latency in the node graph, I absolutely think that needs to be carefully defined and predictable.