Re: Concerning the gap-less output of real-time generated audio in JavaScript

On Mon, Jul 11, 2011 at 6:29 PM, Grant Galitz <grantgalitz@gmail.com> wrote:

>
>
> ---------- Forwarded message ----------
> From: Grant Galitz <grantgalitz@gmail.com>
> Date: Mon, Jul 11, 2011 at 9:23 PM
> Subject: Re: Concerning the gap-less output of real-time generated audio in
> JavaScript
> To: Chris Rogers <crogers@google.com>
>
>
> So we are thinking about a callback based system that allows buffering
> ahead of time, that allows resampling, and uses some form of a buffer ring
> for sample count safety and management optimization? Buffering real time
> without allowing the developer to specify a minimum amount felt like a bad
> plan to be implemented, due to many blocking issues for a callback to be
> launched without major delays (single threaded woes that we need to
> implement APIs around indeed...).
>

I'm not quite sure I follow what you're saying.  But just to clarify what I
meant, I'm proposing *allowing* the developer to specify the "minimum
amount" in the buffering.  So when the hardware drains the buffer to below a
certain threshold (settable by the developer) then the callback will be
fired.  Isn't that what you wanted?

Chris






>
>
> On Mon, Jul 11, 2011 at 3:00 PM, Chris Rogers <crogers@google.com> wrote:
>
>>
>>
>> On Sun, Jul 10, 2011 at 11:11 PM, Grant Galitz <grantgalitz@gmail.com>wrote:
>>
>>> I'll briefly compare the mozilla audio data api and the web audio api and
>>> run through this list of what can be improved upon in web audio.
>>>
>>> - Web Audio does not allow resampling, this is a major thorn in probably
>>> a couple people's butts, because I have to do this in JavaScript manually.
>>> If there is a security concern for bottlenecking, then I'd assume we could
>>> throw in some implementation-side limitations on the number of concurrent
>>> supposed resampling nodes that could be run at the same time.
>>>
>>
>> I agree that it would be useful to allow the creation of AudioContexts
>> with user-settable sample-rates.  It could be as simple as:
>>
>> var context = new AudioContext(sampleRate);
>>
>> where sampleRate must have some kind of reasonable upper and lower bound.
>>
>>
>>
>>>
>>> - Web Audio forces the JavaScript developer to maintain an audio buffer
>>> in JavaScript. This applies for audio that cannot be timed to the web audio
>>> callback, such as an app timed by setInterval that has to produce x samples
>>> every x milliseconds. The Mozilla Audio Data API allows the JS developer to
>>> push samples to the browser and let the browser manage the buffer on its
>>> own. The callback grabbing x number of samples every call is not a buffer on
>>> its own, that's the callback sampling the whole buffer of what I'm talking
>>> about. Buffer ring management in JavaScript takes up some CPU load and it
>>> would always be better in my opinion to let the browser manage such a task.
>>>
>>
>> If sample-rate conversion is taken care of as proposed above, then the CPU
>> overhead of managing a simple ring-buffer in JavaScript should be extremely
>> small and can be implemented in just a few lines of code.  I understand that
>> in your current implementation, you're also dealing with sample-rate
>> conversion which is slower and complicates your code.  But a simple
>> ring-buffer is not very complex.
>>
>>
>>
>>>
>>> - "The callback method knows how often to fire," this is a fallacy, even
>>> flash falls for this issue and can produce clicks and pops on real-time
>>> generated audio (Even their docs hint at this). This is because by the time
>>> the callback API figures out a delay, its buffering may be premature due to
>>> previous calculations and may as a result gap the audio. It is imperative
>>> you let the developer control the buffering process, since only the
>>> developer would truly know how much buffering is needed. Web Audio in chrome
>>> gaps out for instance when we're drawing to a canvas stretched to fullscreen
>>> and a canvas op takes a few milliseconds to perform, to a reasonable person
>>> this would seem inappropriate. This ties in basically with the previous
>>> point of letting the browser manage the buffer passed to it, and allowing
>>> the JS developer to buffer ahead of time rather than having a real-time
>>> thread try to play catch-up with an inherently bad plan.
>>>
>>> - Building up on the last point, in order to achieve ahead-of-time
>>> buffering, I believe it would be wise to either introduce a stub function
>>> that allows samples to be added at any time without waiting for a callback,
>>> just like mozWriteAudio, OR to allow the callback method to be called when
>>> buffering reaches a specified low point *specified* by the developer. This
>>> low point is not how many samples are to be sent to the browser each
>>> callback, but lets the API know WHEN to fire the callback, with the firing
>>> being at a certain number of samples before buffer empty.
>>>
>>
>> I like your second idea of having an internal buffer (in the
>> implementation) whose size can be specified by the developer.  This buffer
>> size is independent of the callback size.  There could also be a mode where
>> this internal buffer can automatically adjust its size depending on runtime
>> characteristics, but this mode could either be enabled or disabled.
>>
>>
>>
>>>
>>> I hope we can use some or all of these points listed in providing a
>>> proper API for real-time generated audio output in JavaScript in a 21st
>>> century browser. :D
>>>
>>
>> I think we can.  My apologies for not yet implementing the ability to
>> choose sample-rates for an AudioContext.  It'll come...
>>
>> Chris
>>
>>
>>
>
>