Fwd: Concerning the gap-less output of real-time generated audio in JavaScript

---------- Forwarded message ----------
From: Grant Galitz <grantgalitz@gmail.com>
Date: Mon, Jul 11, 2011 at 9:23 PM
Subject: Re: Concerning the gap-less output of real-time generated audio in
JavaScript
To: Chris Rogers <crogers@google.com>


So we are thinking about a callback based system that allows buffering ahead
of time, that allows resampling, and uses some form of a buffer ring for
sample count safety and management optimization? Buffering real time without
allowing the developer to specify a minimum amount felt like a bad plan to
be implemented, due to many blocking issues for a callback to be launched
without major delays (single threaded woes that we need to implement APIs
around indeed...).


On Mon, Jul 11, 2011 at 3:00 PM, Chris Rogers <crogers@google.com> wrote:

>
>
> On Sun, Jul 10, 2011 at 11:11 PM, Grant Galitz <grantgalitz@gmail.com>wrote:
>
>> I'll briefly compare the mozilla audio data api and the web audio api and
>> run through this list of what can be improved upon in web audio.
>>
>> - Web Audio does not allow resampling, this is a major thorn in probably a
>> couple people's butts, because I have to do this in JavaScript manually. If
>> there is a security concern for bottlenecking, then I'd assume we could
>> throw in some implementation-side limitations on the number of concurrent
>> supposed resampling nodes that could be run at the same time.
>>
>
> I agree that it would be useful to allow the creation of AudioContexts with
> user-settable sample-rates.  It could be as simple as:
>
> var context = new AudioContext(sampleRate);
>
> where sampleRate must have some kind of reasonable upper and lower bound.
>
>
>
>>
>> - Web Audio forces the JavaScript developer to maintain an audio buffer in
>> JavaScript. This applies for audio that cannot be timed to the web audio
>> callback, such as an app timed by setInterval that has to produce x samples
>> every x milliseconds. The Mozilla Audio Data API allows the JS developer to
>> push samples to the browser and let the browser manage the buffer on its
>> own. The callback grabbing x number of samples every call is not a buffer on
>> its own, that's the callback sampling the whole buffer of what I'm talking
>> about. Buffer ring management in JavaScript takes up some CPU load and it
>> would always be better in my opinion to let the browser manage such a task.
>>
>
> If sample-rate conversion is taken care of as proposed above, then the CPU
> overhead of managing a simple ring-buffer in JavaScript should be extremely
> small and can be implemented in just a few lines of code.  I understand that
> in your current implementation, you're also dealing with sample-rate
> conversion which is slower and complicates your code.  But a simple
> ring-buffer is not very complex.
>
>
>
>>
>> - "The callback method knows how often to fire," this is a fallacy, even
>> flash falls for this issue and can produce clicks and pops on real-time
>> generated audio (Even their docs hint at this). This is because by the time
>> the callback API figures out a delay, its buffering may be premature due to
>> previous calculations and may as a result gap the audio. It is imperative
>> you let the developer control the buffering process, since only the
>> developer would truly know how much buffering is needed. Web Audio in chrome
>> gaps out for instance when we're drawing to a canvas stretched to fullscreen
>> and a canvas op takes a few milliseconds to perform, to a reasonable person
>> this would seem inappropriate. This ties in basically with the previous
>> point of letting the browser manage the buffer passed to it, and allowing
>> the JS developer to buffer ahead of time rather than having a real-time
>> thread try to play catch-up with an inherently bad plan.
>>
>> - Building up on the last point, in order to achieve ahead-of-time
>> buffering, I believe it would be wise to either introduce a stub function
>> that allows samples to be added at any time without waiting for a callback,
>> just like mozWriteAudio, OR to allow the callback method to be called when
>> buffering reaches a specified low point *specified* by the developer. This
>> low point is not how many samples are to be sent to the browser each
>> callback, but lets the API know WHEN to fire the callback, with the firing
>> being at a certain number of samples before buffer empty.
>>
>
> I like your second idea of having an internal buffer (in the
> implementation) whose size can be specified by the developer.  This buffer
> size is independent of the callback size.  There could also be a mode where
> this internal buffer can automatically adjust its size depending on runtime
> characteristics, but this mode could either be enabled or disabled.
>
>
>
>>
>> I hope we can use some or all of these points listed in providing a proper
>> API for real-time generated audio output in JavaScript in a 21st century
>> browser. :D
>>
>
> I think we can.  My apologies for not yet implementing the ability to
> choose sample-rates for an AudioContext.  It'll come...
>
> Chris
>
>
>