Re: Sample-accurate JS output (was: scheduling subgraphs)

Thanks Chris!

Your post prompts another thought: one more reason to pass the buffer  
size into a JavaScriptAudioNode via AudioProcessingEvent (as opposed  
to letting the node dictate) is that it could be useful to tune the  
AudioContext's behavior as a whole, for low latency vs. high  
stability.  This setting on the AudioContext -- whatever form it  
happens to take -- would then drive the buffer size and batching  
behavior of the whole engine, in a top-down kind of way.  So I think  
this is a highly desirable feature to add.

For example Noteflight values playback robustness over latency --  
there is little or no real-time control of sound output.  However,  
other applications will have a completely different profile and might  
want really low latency even if there's some risk of jitter or buffer  
underrun.  And many applications would want to occupy some kind of  
middle ground.

The Flash Player 10 audio API supports a concept of globally  
requesting low latency vs. playback stability by a very quirky  
approach that I wouldn't want to see us emulate, but the end result is  
that one winds up indirectly specifying a "batch size" for audio  
buffers between 1K and 8K samples.  StandingWave then simply picks  
this batch size up from Flash and propagates it through its own code.

...Joe

On Oct 21, 2010, at 1:29 PM, Chris Rogers wrote:

> Hi Joe,
>
> Thanks for the very detailed description.  Interestingly, this is  
> effectively what I'm already doing for AudioBufferSourceNode  
> internally, minus the batching and threading stuff.  The number N in  
> the current engine is 128 @44.1KHz (for low latency) so I think this  
> would be too small of a batch size to dispatch periodically to the  
> main thread.  But, this can be easily solved by buffering into  
> larger chunks, which I'm already doing in my current  
> JavaScriptAudioNode.
>
> It would be good to have the API for the generator (output-only) and  
> the processor (input and output) case be very close or the same,  
> even if this optimization is only generally possible for the  
> generator.  Currently I have a JavaScriptAudioNode to handle both  
> cases...
>
> Anyway, I really appreciate your great insights and experience here!
>
> Cheers,
> Chris
>
>
>
> On Wed, Oct 20, 2010 at 3:45 PM, Joseph Berkovitz  
> <joe@noteflight.com> wrote:
> Further implementation thoughts on this issue -- this should address  
> the many-short-notes cases as well as other pathological cases.
>
> When I say "JS nodes" here, by the way, I am only talking about  
> *generator* JS nodes, i.e. JS nodes with no inputs.  I don't have  
> any good ideas about JS nodes that act as filters, I think if one  
> has a lot of those one may be inherently hosed in terms of  
> performance.
>
> The goal is to restrict JS activity to only those JS generator nodes  
> which can contribute output to a synchronous processing batch, and  
> to pad each node's output on either side as needed to fill out its  
> buffers to the size expected by the audio engine.  Each node only  
> "sees" a request for some # of samples at some specified start time  
> as specified in the AudioProcessingEvent, and doesn't have to worry  
> about padding or about being called at an inappropriate time.
>
> 1. In general do not allow JS nodes to determine their own buffer  
> size.  Provide a event.bufferLength attribute in  
> AudioProcessingEvent which JS nodes will respect: they are expected  
> to return buffer(s) of exactly this length with the first sample  
> reflecting the generated signal at event.playbackTime.  Dispense  
> with the ability to specify a bufferLength at JS node creation time;  
> the audio engine is in charge, not the programmer.
>
> 2. (rough outline of algorithm, ignoring threading issues -- idea is  
> to context-switch once and process all JS generator nodes in one gulp)
>    let N be number of samples in a synchronous processing batch for  
> the audio engine (i.e. a graph-wide batch pushed through all nodes  
> to the destination)
>    let batchTime be the current rendering time of the first sample  
> in the batch
>    let startTime, endTime be start, end times of some JS generator  
> node (i.e. the noteOn/startAt() or noteOff()/stopAt() times)
>    consider a node active if the range (batchTime, batchTime +  
> (N-1)*sampleRate) intersects the range (startTime, endTime)
>    dispatch an AudioProcessingEvent to such a node, where the  
> event's playbackTime and bufferLength together describe the above  
> intersected range (which will usually be an entire processing batch  
> of N samples).  The result may be less than N samples, however, if  
> the node became active or inactive during the processing batch.
>    left-pad the returned samples by (startTime - batchTime) /  
> sampleRate, restricting to range 0 .. N
>    right-pad the returned samples by N - ((endTime - batchTime) /  
> sampleRate) restricting to range 0 .. N
>
> I didn't make this algorithm up from scratch, it's adapted from the  
> StandingWave Performance code, so I believe it pretty much works.
>
> ... .  .    .       Joe
>
> Joe Berkovitz
> President
> Noteflight LLC
> 160 Sidney St, Cambridge, MA 02139
> phone: +1 978 314 6271
> www.noteflight.com
>
>
> On Oct 20, 2010, at 3:27 PM, Chris Rogers wrote:
>
>> Yes, that's what I've been thinking as well.  There's still the  
>> buffering/latency issue which will affect how near into the future  
>> it will be possible to schedule these types of events, but I  
>> suppose that's a given.  Also, there could be pathological cases  
>> where there are many very short notes which aren't exactly at the  
>> same time, but close.  Then they wouldn't be processed properly in  
>> the batch.  But, with the proper kind of algorithm, maybe even  
>> these cases could be coalesced if great care were taken, and  
>> possibly at the cost of even greater buffering.
>>
>> Chris
>>
>> On Wed, Oct 20, 2010 at 12:51 PM, Joseph Berkovitz <joe@noteflight.com 
>> > wrote:
>>
>> Implementation thought:
>>
>> I was thinking, if all JS nodes process sample batches in lock  
>> step, can all active JS nodes be scheduled to run in sequence in a  
>> single thread context switch, instead of context-switching once per  
>> node?
>>
>
>
>

... .  .    .       Joe

Joe Berkovitz
President
Noteflight LLC
160 Sidney St, Cambridge, MA 02139
phone: +1 978 314 6271
www.noteflight.com