Re: Music Synthesis

On Wed, Jun 16, 2010 at 5:41 PM, Robert O'Callahan <robert@ocallahan.org>wrote:

> Can you be more specific about the sorts of audio processing algorithms
> that would be hard to expose in a library that works on sample arrays?


By library that works on sample arrays, I'm assuming a vector library
similar to vecLib.framework.

- Dynamics processors of any kind, compressors, limiters, expanders,
including multi-band versions
- Chorus and other pitch type effects, phasors, flangers, etc.
- time-stretching and pitch shifting algorithms
- certain kinds of synthesis algorithms which incorporate dynamic envelopes
and LFOs
- guitar amplifier simulation
- etc. etc.



>
>
> On Thu, Jun 17, 2010 at 11:34 AM, Chris Rogers <crogers@google.com> wrote:
>
>> Also, I believe it's very important to provide a useful, high-level
>> javascript API that's dependable, scalable, and powerful.  Simply providing
>> a library of vector calls, and then hoping that javascript will be fast
>> enough to take up the slack in a worker thread just doesn't seem realistic.
>
>
> It would be better to be specific. From experience we don't trust people's
> intuitions about what is or isn't possible in Javascript.
>

OK, fair enough, but I can ask the same of you.  You're proposing that we
make an important design decision which hasn't yet been backed up with
working code.  The Mozilla demos I've seen so far are doing fairly
rudimentary processing and have very noticeable latency issues.  I know
you're aiming to correct some of these problems by using web workers, but we
don't have that implementation yet.  It would be helpful to see the system
work well under high stress loads doing some fairly demanding rendering,
like convolution and 3D spatialization.  Some compute resources will still
be needed for game physics and graphics.


> It would require all browser vendors to implement worker threads and
>> javascript runtimes with real-time constraints in mind to guarantee good
>> latency performance and glitch-free operation.  This is extremely difficult
>> and is a lot to expect from all the different browser vendors.
>
>
> We're going to have to go in that direction anyway, for WebGL gaming. (I
> assume that no-one's going to suggest we support an entire 3D
> first-person-shooter engine API in browsers.) We won't get to hard realtime
> guarantees, but we should be able to get pretty good results.
>

The real-time demands for graphics and audio are very different things.  For
graphics and video, it's tolerable to lose a frame here and there.  But for
audio, missing even a single sample can result in a very objectionable
click.


>
> I guess that's a scope question: do we need to have hard latency guarantees
> so you can run a performance-quality music synthesizer in a Web browser? I
> don't think we do. Not in version 1 anyway.
>
> (If we do need to have such guarantees, this is still "extremely difficult
> and is a lot to expect from all the different browser vendors" whatever the
> API looks like.)
>
> Rob
>

We should have at least tolerable latency performance for simple music
applications and games.  The system I'm proposing has fairly good
performance characteristics and already has a preliminary open-source
implementation.  I'm very open to discussing how the code works and
integration into Mozilla's codebase.  I already have it working in two
browsers.

I'm trying to provide a solution which satisfies your needs for doing direct
javascript generation and processing, but also offers other capabilities.  I
hope you will consider the usefulness of what I'm offering.

Best Regards,
Chris