Re: Requirements for Web audio APIs

On Mon, May 23, 2011 at 2:54 PM, Robert O'Callahan <robert@ocallahan.org>wrote:

> On Tue, May 24, 2011 at 5:42 AM, Chris Rogers <crogers@google.com> wrote:
>
>> Rob, I'm afraid I'll just have to strongly disagree.  The Stream API is
>> designed specially for peer-to-peer communication.  Superficially it has
>> sources and sinks of data like the Web Audio API, but if you really read
>> both specs carefully their design goals are quite distinct. The Web Audio
>> API has had a great deal of care in its design to address a variety of use
>> cases for client side games, music, and interactive applications.  The
>> Stream API does not replace its functionality.  Does it have the concept of
>> intermediate processing modules, routing graphs, implicit unity gain summing
>> junctions, fanout support, channel mix-up and mix-down, sample-accurate
>> timing, among many other design features? -- no it does not.
>
>
> Indeed. And the Web Audio API does not have support for capturing local
> audio, recording and compressing processed audio, or sending processed audio
> over the network (and we have only a sketch for how it might handle A/V
> synchronization). We need fix to these deficiencies in both specs by
> bringing them together somehow.
>

Capturing local audio is simply a matter of getting the user's permission.
 It's straightforward to have an AudioNode for this source which can blend
seamlessly into the current processing architecture in the Web Audio API.
 A/V sync is clearly something to be handled at the HTMLMediaElement API
level and you're free to propose an API and describe a plausible
implementation if you like.  Recording/mixdown is something which can
currently be done via a JavaScriptAudioNode and the file API.  Compressing
processed audio could be done with an API like this:

var arrayBuffer = audioBuffer.createCompressedAudioData(audioFormat);
// Then send arrayBuffer with XMLHttpRequest

Sending processed audio over the network (in the peer-to-peer communications
case) is something still to be worked out in the details.  But I wouldn't
assume that the Web Audio API and the Stream API can't be reconciled in a
reasonable way.


>
> If we persist with the approach that Web Audio handles one set of use-cases
> and Streams handle a different set of use-cases, we'll be making life
> unnecessarily difficult for authors whose use-cases need features from both
> specs. See for example the scenarios here:
>
> https://wiki.mozilla.org/MediaStreamAPI#Scenarios
> At least #3, #4, #5, and #6 require functionality that is currently split
> across both proposals --- and I'm sure people can think of more creative
> ways to mix and match features.
>
> These are both only proposals. Now is the time to make changes to avoid
> future problems. This is why we have a standards process.
>

I and others involved in the process have already spent a great deal of time
trying to make sure the Web Audio API specification is detailed and coherent
enough to form the basis for an audio architecture.  Most parts of the
proposal are actually implemented, and the parts which are not I'm able to
explain in quite a lot of technical detail how they will or can be
implemented.  This includes such unimplemented features as audio input
(capturing local audio).  I'm not sure that your proposal is fleshed out in
nearly the kind of detail which is necessary to know if it would work or
even how to begin implementing it.  It's really a very rough sketch at this
point with lots of hand-waving and I think it would take a very long time to
refine it to the point where it could form a useful basis as a
specification.  I say this with all due respect to the time and
consideration that you've put into it so far, but I still hope that you will
consider the Web Audio API proposal as a good starting point.

Regards,
Chris