Re: Web Audio API Proposal

First, I apologize for the unnecessarily strident tone of my first email.

On Wed, Jun 16, 2010 at 1:36 AM, Chris Marrin <cmarrin@apple.com> wrote:

> I believe the two proposals mark the limits (simple to complex) of what is
> needed for an audio processing API in a web browser. And I think that is an
> excellent starting point. The Mozilla proposal would severely limit the
> types of audio processing possible on many devices, especially mobile
> devices. Providing native APIs for the most common processing models makes
> as much sense as providing filters in SVG or spline curves in Canvas.
>

Filters in SVG are an interesting analogy. The fixed set of primitives is
very limiting, even though you can combine them in ways similar to the
proposed "audio node graph" ... you often end up doing horrible hacks to get
the effects that you want. Those hacks often require many primitives so
performance ends up being poor. Similarly, GL moved away from the
fixed-function pipeline to a much more flexible and programmable model.

Splines in 2D canvas are not as good an analogy because they are a simple
abstraction that meets practically all path construction needs. I haven't
seen people trying to hack around the limitations of splines.

Importantly, these nodes DO NOT give access to the samples themselves. I
> believe this is important because it allows very efficient audio processing
> chains to be created and optimized without the need to expose the underlying
> details of how buffering occurs.
>

This is a key point. Can you describe how an implementation would exploit
that restriction?

I believe section (3) is a very important one to get right. I see 2 problems
> with leaving FFT processing to JavaScript. First of all, an FFT is such a
> standard algorithm it seems like a very reasonable and obvious thing to
> include in an API.
>

Maybe so, but we could make it part of a built-in math library that works on
typed arrays. Then it could be used for all kinds of applications.

Second, there are some implementations of JavaScript that will not be able
> to keep up with the processing of 48Khz stereo audio. These implementations
> will have to reduce the sample rate,which will make the FFT calculations
> less accurate. And even if an implementation is able to keep up with the
> data rate, it will leave very little for any other JavaScript to run at any
> sort of reasonable frame rate.
>

So these must be platforms where "native" FFTs are much faster than any
possible JS implementation. What accounts for that performance difference?

Rob
-- 
"He was pierced for our transgressions, he was crushed for our iniquities;
the punishment that brought us peace was upon him, and by his wounds we are
healed. We all, like sheep, have gone astray, each of us has turned to his
own way; and the LORD has laid on him the iniquity of us all." [Isaiah
53:5-6]