Re: Web Audio API Proposal

On Jun 15, 2010, at 4:38 PM, Robert O'Callahan wrote:

> On Wed, Jun 16, 2010 at 11:01 AM, Chris Marrin <cmarrin@apple.com> wrote:
> It's an interesting idea to imagine an "audio processing language" for use here. But I don't think JavaScript is the appropriate candidate for such a language. OpenCL would do a good job of audio processing. Someday perhaps WebGL shaders will take the place of SVG filters. And perhaps in the future we will have WebCL, which can bring OpenCL capabilities to the browser. At that point, "programmable audio processing" might be possible. Until then, I think we need a set of fixed function audio processing capabilities.
> 
> Why not plan for that future now? Otherwise we'll end up saddled with a vestigial fixed-function pipeline, much like OpenGL has now.

How does a JavaScript-only audio processing system plan for the future? Currently WebGL is an API that is similar to OpenGL ES 2.0. This includes support for programmable vertex shaders and fragment shaders because this is the commonly available set across many hardware platforms. But newer platforms include Geometry Shaders, which is a very new technology but one that promises incredible performance gains in the future. If we were to wait for Geometry Shaders to release WebGL, it would be another 3 years out, at least. We've planned for the future by accommodating updates in WebGL functionality through extensions and versioning of the API.

Planning for the future means providing an infrastructure now into which programmable audio units can be added later. I believe Chris' architecture does this. We may want to minimize the set of "fixed-function" nodes, but I think we need some.

> 
> I disagree with the notion that SVG filters inevitably lead to horrible hacks. You can do very interesting things with chains of SVG filters. You can abuse them, too, and pay a hefty penalty for your sins.
> 
> I claim that stringing together a series of SVG filters to achieve the effect you want is often more clumsy and lower performance than writing the effect directly in a more programmable model --- even if you find the simplest possible set of filters that work.

What more programmable model is available to test this statement against? Often the effect you're looking for is available in a single filter (like blur), just like often the audio effect you're looking for is available in a single node (like echo). And because of the way SVG filters are defined, it's possible for an implementation to combine filters into a single programmable shader without the author having to do any work. Similarly, an implementation can evaluate an audio filter chain and optimize it under the hood.

> 
> > Splines in 2D canvas are not as good an analogy because they are a simple abstraction that meets practically all path construction needs. I haven't seen people trying to hack around the limitations of splines.
> 
> Many of the audio nodes described in Chris' proposal are similarly simple abstractions of existing audio functionality.
> 
> OK, but is there any evidence that they're complete? What fraction of use cases would be forced to resort to JS-based sample processing?

We can't know that. What we can know is that there is a large body of experience in existing audio processing libraries where we can draw a set of commonly used primitives.

> 
> I have never seen anyone implement their own kinds of curves in 2D canvas because Beziers weren't adequate to describe their paths.

I don't want to belabor this point, but there are many mathematical curve types that can't be represented with cubic beziers. These are usually approximated with piecewise linear curves. 

-----
~Chris
cmarrin@apple.com

Received on Wednesday, 16 June 2010 00:11:57 UTC