Re: Questioning the current direction of the Web Audio API

On Mon, Oct 21, 2013 at 9:54 AM, Robert O'Callahan <robert@ocallahan.org>wrote:

> [Let me note for historical purposes that there was a window of
> opportunity to promote a more JS-aligned solution over Web Audio, but we
> missed that window. That was partly my fault for not getting an
> implementation of MediaStream Processing into Firefox early enough, but I
> also had practically no support in this group helping me push back against
> the Webkit "this is what we're shipping" juggernaut.]
>

Practically yes, although I feel that I tried to push for it to the point
of annoying the bugger out of the group. I'm not quite sure what was so
awesome about this API that made it worth fighting for so fiercely. But
that ship sailed, although I did try to pull MSP up even after we had
decided to bury it, again not to a very warm welcome.


> It's important to keep in mind when comparing Web APIs to your favourite
> native audio framework that on the Web, we have to handle untrusted
> (possibly malicious, very likely poorly written) JS code. That's one reason
> why running user JS audio processing code on a realtime audio processing
> thread is a rather trickier proposition for us than for your native audio
> framework. (And please don't play the "but MY code is perfect" card, it's
> not helpful.)
>

Native or not, letting people do anything on a high priority thread has its
risks. Not as high risks as letting people run arbitrary code on the GPU,
but risky anyway, e.g. you can pretty easily brick (until battery removal)
a single core device by just having a convolution node with massive uneven
kernels.

Actually security is an argument in favor of a minimal API, because the
less we specify, less attack vectors we have likely created. This is
because unfortunately spec writers don't write any less imperfect stuff
than anyone else.


> So where can we go from here? I think we pretty clearly need a version of
> ScriptProcessorNode that does JS audio processing off the main thread. The
> obvious and expedient approach would be to have it dispatch messages to a
> Worker just like MSP did. It should work with Float32 typed arrays rather
> than AudioBuffers and be designed to avoid requiring buffer copies. I think
> that we can arrange for very-short-running JS audio processing to run
> synchronously during Web Audio processing (by having that realtime thread
> wait for the JS processing to complete, with a short timeout in case it's
> slow). We need to define what happens if the processing is too slow;
> ideally that should just cause a latency increase, if we're not in a true
> overload situation.
>
> That should help a lot for a lot of use-cases. If it turns out to be an
> incomplete solution, we'll have a lot more data to inform our next move.
>

I highly agree with this. In my opinion, this is the sane approach of
solving problems little by little, gathering data of the real pain points
as we go, and pivoting as needed.

I want to clarify to everyone that I'm not saying that "no native nodes,
ever". In fact I'm most of the time an obsessed performance-junkie, which
has backlashed at me oh-so-many times, but at least so far my wounds
indicate the same things they always say in every performance 101: Don't
Optimize Prematurely.

What I'm saying is that native nodes should've come not in v1 or possibly
even v2, but once we had perfected the custom processing. Only then we
could've started to see the real patterns out in the wild and started to
optimize those. Now we just defined arbitrary "basic" building blocks (one
could argue against for example convolution being one of them, given that
even popular DAWs are lacking implementations), imagined what the use cases
for those would be and optimized around them. Premature optimization (and
premature abstraction, which is even worse) at its best, in my opinion.

Sorry for throwing more fuel at the fire, but I think it's worth having a
little bit of retrospective about things like these every once in a while.
:)

Cheers,
Jussi

Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w  *
> *
>