Re: Web Audio API Spec

On Mon, Jun 20, 2011 at 7:51 PM, Ian Ni-Lewis <ilewis@google.com> wrote:

> Responses inline...
>
> On Mon, Jun 20, 2011 at 1:06 AM, Jussi Kalliokoski <
> jussi.kalliokoski@gmail.com> wrote:
>
>> First of all, the spec is way too non-specific. For example, if the spec
>> says "a more advanced implementation could", that just gives too much
>> freedom to implementations, and can result in serious pain to web
>> developers. Say, some implementation decides not to implement a delay node,
>> so if your application requires that behaviour, you have to have a pure js
>> implementation and feature detection in place to counter that, and that just
>> beats the purpose of having a higher level API.
>>
>
> I think this is a valid point, if what you're asking for is a minimum set
> of supported nodes. Implementations should be allowed to support additional
> nodes through the same extension mechanisms that are currently used in other
> specs such as HTML and CSS.
>
>
>>
>> The next thing is somewhat the same thing, this is also something that
>> I've confirmed in my discussions with people. Currently the spec doesn't
>> specify the exact algorithms to use with the processing of the nodes.
>>
>
> Very good point. We should specify this.
>
>
>> Also, the current spec is very much wrong in a way that... It let's the
>> implementation mandate the spec and not vice versa, and that is just not
>> acceptable from my point of view, and that is also confirmed in my
>> discussions. An example of this is that when you create a pure javascript
>> processing thingy, as far as I know, you can't know for sure that you would
>> get the buffer size and sample rate that you asked for.
>>
>
> This doesn't seem like the implementation mandating the spec; it seems like
> an acknowledgement of varying target HW platforms. To give another example,
> the HTML spec allows the "implementation" to "mandate" the size of the
> display. I don't think this is because the spec is being heavy-handed; I
> think it's because in the real world people run their browsers at many
> different widths and heights, and it's useful for developers to know at what
> resolution the user has chosen to view the page. It's possible, of course,
> to let the web page mandate a logical resolution and force the browser to
> resample the page to fit in the user's window, but that seldom produces
> optimal results.
>

There are more examples of implementation mandating the spec:
http://chromium.googlecode.com/svn/trunk/samples/audio/specification/specification.html#dfn-noteOff
,
but my main point is that you should be able to do it, sub-optimal or not,
for example if you're rendering audio into a file, performance is not as
important as the end result.


> I think this is also not a very good idea, if we have a higher level API
>> like this, it should be able to do the resampling for you, otherwise you may
>> have to do it in JS, which is a performance hit and often a nasty hack. I
>> think that the spec should state that IFF the sample rate and buffer size
>> aren't specified, the implementation tries to find the optimal values for
>> them.
>>
>
> What puzzles me about this argument is the assumption that the JS would
> need to resample. Most of the processing I'd envision doing is more or less
> sample rate independent, in the sense that the sample rate is a free
> variable in the processing function. It's trivial, for instance, to generate
> a sine wave or perform filtering at any arbitrary sample rate.
>
> Buffer size is a little less flexible, given that you may have for instance
> an FFT that requires a specific buffer size to operate. But rebuffering the
> data isn't nearly as expensive as resampling, and doing it in JS doesn't
> seem like a particularly bad idea.
>
> On the other hand, if we allowed JS to demand a specific sample rate, then
> the implementation would be forced to resample twice--once on the way in,
> once on the way out--and perform extra buffering. This imposes a cost
> without a clear benefit. In fact, it's quite possible that quality would
> suffer.
>

It's even more inefficient when done in JS, and having the feature would be
consistent with the purpose of the audio processing nodes being native code
to improve performance. And I'm not saying that these parameters should
always be decided by the developer, but in the case the developer needs a
certain sample rate or buffer size, I think he/she should get them, instead
of being forced to the optimal values. But yes, buffer size is more or less
trivial, my main concern is the sample rate.


Perhaps you could clarify the sorts of processing you had in mind, where JS
> would be unable to deal with arbitrary sample rates and buffer sizes.
>

One important use case is jsmad ( http://www.jsmad.org ), that tries to
implement mp3 codec in browsers that it doesn't work in. It does so by using
the audio APIs for playback and real time decoding, which in itself is quite
performance hogging, not to mention adding javascript resampling. Related
bug can be found here:
http://code.google.com/p/chromium/issues/detail?id=86767 . They're receiving
quite a lot of attention currently, so I think the issue needs attention
soon.

Cheers!
Jussi


> Thanks
> Ian
>

>
>> I hope that we can raise a good discussion regarding these problems.
>>
>> Jussi
>>
>
>
>
> --
> Ian Ni-Lewis
> Developer Advocate
> Google Game Developer Relations
>
>