Re: Making Web Audio APIs More Competitive with native

Hi Chris,

I do appreciate your proposal. It makes sense *and* it has an
implementation. My primary question is whether the benefits of
binding to or building on top of an existing (or upcoming) API
proposal have been weighed against those of a new api for the web.
I'm not specifically pushing this api or that, especially at this
stage. I'll just leave this now at "it has been brought up".

Rgd OpenSL ES, let me just say this - after spending a few
days actually letting the spec sink in, I wonder why Khronos
bothered with a new *more complicated* API given that OpenAL
already exists, unless I'm missing some point about the special
API design. The DLS synth and MIDI may have been the
sole motivations over and above OpenAL .. but I digress.

Some specific answers -

>> 1) Reducing developer cognitive load in the long run,
>
> I'm not really sure what this means.

It is nice to be able to learn an audio model and apply
it on a few different platforms (the web being one) without having
to learn a new one for each. Helps code portability too ... as
opposed to a redesign.

>> 2) Maximizing audio hardware access in the long run and
>
> Browsers already have access to the basic audio hardware to support <audio>
> and <video> elements.

If you have a sound chip with great 3D spatialization (obstacle support
for instance), it'll be nice to be able to use that. If you have multichannel
audio out, it'll be nice to access them, etc.

> Something you may not realize is that Android, iOS, and other major
> mobile platforms use WebKit for their web implementation.

I do. That is largely why I'm here.

> One point which is important to realize is that OpenSL is a low-level C API
> which is difficult and complex to work with, even for the simplest audio
> tasks.  For examples of this complexity, please see the numerous code
> examples in the OpenSL specification.

The code examples do look complicated, but that's the C nature kicking in.
The C code compares to what programming COM looks like in C, versus
say in python. I've stared at the code long enough to say that .. and I've
had the professional misfortune of having to deal with (a little) COM in C :)

> To be fair, it does support DLS sample-banks and a
> built-in sequencer for MIDI files.  But this is a feature which can be
> handled in other ways in my API.

Note that this is a clear case where you can just bring that feature
in without having to develop a DLS synth module or the equivalent,
or having to ship a bank with webkit (hopefully).

Regards,
-Srikumar

On Sun, Dec 12, 2010 at 5:00 AM, Chris Rogers <crogers@google.com> wrote:
> On Fri, Dec 10, 2010 at 8:39 PM, Kumar <srikumarks@gmail.com> wrote:
>>
>> Hi Chris,
>>
>> Yes, you're right about the immaturity of OpenSL ES. It is a
>> relatively recent spec indeed. OpenAL (together with EFX) is
>> a more mature candidate.
>
> I implemented the underlying engine for OpenAL on Mac OS X.  Beyond the very
> base-level API, there are significant differences in the features and even
> the API between platforms.  With help from WebKit engineers at Apple and
> others on this list, I've tried to make my proposed API a somewhat more
> compelling alternative which supports a superset of the features of OpenAL,
> while addressing the current serious compatibility problems it has.  One
> problem it has is the way that environmental spaces (room effects) are
> handled.  This same issue also exists with OpenSL.  I wrote about this in
> section 12 of my specification (Linear Effects using Convolution -
> subsection "Motivation for use as a Standard"):
> http://chromium.googlecode.com/svn/trunk/samples/audio/specification/specification.html#Spatialization-section
>>
>> The symmetry I was referring to is more about the broad
>> principles behind considering existing APIs ... three of them -
>>
>> 1) Reducing developer cognitive load in the long run,
>
> I'm not really sure what this means.
>
>>
>> 2) Maximizing audio hardware access in the long run and
>
> Browsers already have access to the basic audio hardware to support <audio>
> and <video> elements.
>
>> 3) Reducing browser implementation burden in the long run.
>>
>> On various platforms, we have OpenAL, Core Audio,
>> DirectSound, OSS, ALSA, ESD, JACK, LADSPA. Of these,
>> possibly OpenAL has the widest base. If we're to look a couple
>> of years into the future, I can't help wonder if OpenSL ES
>> will be the more established native API. It is certainly getting
>> on to Android [http://developer.android.com/sdk/ndk/overview.html].
>>
>> If OpenSL ES is prevalent on mobile platforms and it can
>> wrap around existing platform-specific audio APIs and codecs,
>> then having its object model directly available via Javascript
>> seems likely to maximize the audio hardware access across the
>> board. Given that platform vendors will provide the implementation,
>> adopting it will reduce the implementation burden on WebKit/Mozilla,
>> and reduce the need for developers to work with one more audio API
>> in the mix.
>
> Kumar, I think I simply disagree that OpenSL would be the path of least
> resistance to implement an audio API in the browser.  As far as WebKit is
> concerned, I've just landed a complete implementation of my API which is set
> to work with Apple's Safari and Google's Chrome.  Something you may not
> realize is that Android, iOS, and other major mobile platforms use WebKit
> for their web implementation.  Thus there is substantial finished work that
> potentially could be leveraged there.  Having looked over the complexity of
> the OpenSL specification and having experience in the WebKit codebase
> myself, I believe it would be easier to adopt my implementation.  As far as
> other browser codebases go, I don't think there's any compelling evidence
> that an OpenSL implementation would be easier.
> One point which is important to realize is that OpenSL is a low-level C API
> which is difficult and complex to work with, even for the simplest audio
> tasks.  For examples of this complexity, please see the numerous code
> examples in the OpenSL specification.  The style and low-level nature of the
> API also would, in my opinion, make it very difficult to adapt to a
> JavaScript version of the API, with major changes being required.  In my
> proposed API, great care has been taken to make it as powerful as possible,
> while reducing the complexity of the code which JavaScript developers have
> to write.
> Lastly, despite the complexity of OpenSL (their specification is 555 pages
> long!), it does not deliver all of the features of my API.  Their API does
> not support fully modular routing, and instead has a rather limited fixed
> single-send architecture.  It is not easily scalable due to its limited
> modularity.  Also, it supports a rather crude model for spatialization and
> environmental effects.  To be fair, it does support DLS sample-banks and a
> built-in sequencer for MIDI files.  But this is a feature which can be
> handled in other ways in my API.
> Chris
>
>

Received on Sunday, 12 December 2010 06:49:49 UTC