Re: Questioning the current direction of the Web Audio API

> ScriptProcessorNode is perfectly equivalent functionally to what Firefox
provided back then .. and we're still complaining about it

But if there was only this piece functionality to work on, it would
probably be much cleaner and efficient than it is today.

> The current broken-ness of the ScriptProcessorNode [...] stems from the
fact that running JS code [...] introduces intolerable communication
latencies and potential for glitches for engaging sound.

Once again, that's a problem that could have been tackled if the
specification was focusing on it rather than re-inventing the wheel.

Guys from RjDj (http://rjdj.me/) have compiled big Pure Data patches to a
single dsp function using emscripten and asm.js, and their benchmarks shown
that they have ***same performance in a ScriptProcessorNode than Pure Data
on the desktop*** which to me is quite impressive.

>From my experience, I have used WebPd - which I haven't optimized at all -
rather successfully in performances, without a single glitch with rather
big patches (~ 200 objects) on crappy laptops.

If optimized JavaScript + ScriptProcessorNode is as performant as Pure Data
on the desktop, this would prove that you can do pretty much anything you
need in there. So once again, I don't see the need for native nodes. In
fact an approach that would have been more clever in my opinion is a set of
native dsp functions (possibly implementing the same functionality as WAA),
which you could compose freely together with plain JavaScript functions.




2013/10/21 Srikumar Karaikudi Subramanian <srikumarks@gmail.com>

> Then, given the current limitations of the ScriptProcessorNode, if I
> implemented just the missing parts with ScriptProcessorNodes, I would end
> up being in a worse place in terms of performance than if I had gone with
> just doing everything in a single ScriptProcessorNode, due to reasons
> already mentioned by Sebastien.
>
>
> This is pretty much what Lonce and I concluded - if you want to use the
> ScriptProcessorNode, do everything in it, or use native nodes exclusively.
> While the latter approach is quite viable for applications like games
> (though perhaps not for truly experimental game audio), the former is only
> viable in situations where audio takes centre stage ... but at least both
> are *viable* to a good extent today. The ScriptProcessorNode is simply
> *not* something that can be used to emulate the other nodes.[1]
>
> I would like to step back a bit and look at the "hindsight" aspect of this
> discussion.
>
> Way back, we *did* have a "minimum viable product" specd more or less as
> you ask - "easy way to write sound, any sound, to the users' speakers".
> Before the Web Audio API's early incarnations even existed (iirc), Firefox
> already had such an experimental API available which was already getting
> some visibility. The problem was that it was "minimal", but not really
> "viable" for many applications. In fact, what we have today in the
> ScriptProcessorNode is perfectly equivalent functionally to what Firefox
> provided back then .. and we're still complaining about it.
>
> The current broken-ness of the ScriptProcessorNode is not entirely a
> failure of the WG. It stems from the fact that running JS code in any of
> the sandboxed environments available in the browser - the main thread or
> workers - introduces intolerable communication latencies and potential for
> glitches for engaging sound. In short, these sandboxed environment suck for
> complete programming flexibility. Will communication with workers improve?
> Perhaps. Should the WG push that? Maybe. Will it happen across the board on
> all devices? I Don't know. Would devices be able to run some efficient
> native audio components? Hell yes, they already do. The design flaw, from
> this perspective, looks like not enough native components.
>
> The limitations of the pure JS approach reared its ugly head very early on
> and continue to persist till date. If we coded a game with that API,
> detected a game event and triggered a sound now, we hear it tomorrow,
> unless you're ok with audio breaking up. If we looked around at other
> flexible systems such as SuperCollider / MaxMSP / PD / CSound, we'd notice
> that all of them had gone through building components in native code for
> performance and plugging them into a glue language - with some more
> expressive and flexible than others. If we consider the emerging devices,
> JS performance (compute per watt) was expected to remain low on them for a
> while ... and is still relatively low on ARM devices *today* with no clear
> future other than accelerating various aspects one by one using native code
> or the GPU. I personally had to make a choice back then whether to back the
> pure-JS API or the Web Audio API and I chose to commit to WAA simply
> because it had folks who were as concerned about minimizing latency and
> glitches as I was/am.
>
> As for what might've been a *viable* product, what I might've tried first
> is to take the SuperCollider server, which is pretty efficient, and build a
> JS front end for it. You might've gotten plenty native nodes to keep you
> busy and enough programming flexibility, thanks to JS.
>
> -Kumar
>
> [1] "Taming the ScriptProcessorNode"
> http://sriku.org/blog/2013/01/30/taming-the-scriptprocessornode/
>
> On 20 Oct, 2013, at 10:35 PM, Jussi Kalliokoski <
> jussi.kalliokoski@gmail.com> wrote:
>
> I'm a bit late for the party here, but... I'm sure no one is surprised
> that I *very very* much agree with Sebastien here. In fact I had probably
> been in the working group for just a few weeks before starting to complain
> about the (to me) completely backwards way of building this API. In my book
> the Web Audio API has become yet another warning example in the wall of
> shame for The Extensible Web Manifesto<http://extensiblewebmanifesto.org/>,
> where what people needed was the minimum viable product, i.e. easy way to
> write sound, any sound, to the users' speakers, and they needed it direly.
> Instead they got more than two years of waiting (depending on how you count
> of course, you could say that the Webkit implementation more shortly, but
> you could also say that we haven't reached v1 yet) just to get a monolithic
> framework that's hard to extend to their needs.
>
> I've given the API my fair share of shots, trying to use it for both games
> and music, for cases where in theory the basic building blocks provided by
> the API should be enough (with a few hacks like looped "noise"), for
> example jet engine simulation as well as woodwind simulation. Every time
> eventually I've had to give up due to some limitation in the *framework* (such
> as circular routing not being possible without a 128 sample delay in the
> loop) or the some of the *nodes* themselves. Then, given the current
> limitations of the ScriptProcessorNode, if I implemented just the missing
> parts with ScriptProcessorNodes, I would end up being in a worse place in
> terms of performance than if I had gone with just doing everything in a
> single ScriptProcessorNode, due to reasons already mentioned by Sebastien.
>
> We were also hitting the same issues at ofmlabs. In fact, in the
> discussions I've had with my colleagues even outside ofmlabs, anyone who
> has been in longer term contact with the API shares the frustration (or
> maybe they're just nice to me when I'm ranting :).
>
> All this said, I'm sure most, if not all, of us here more or less see the
> issues now and I'm glad we're moving to first fix the gaping awful holes in
> the API for v1, and for v2 move on to what we should have started with:
> making the ScriptProcessorNode not just an escape hatch or a complement to
> the native nodes, but the core of the API on which to build on. Now, I know
> that hindsight is easy, but if we had started with just the
> ScriptProcessorNode two years ago and started getting developers to build
> around it, then optimize and build on the patterns they form, we wouldn't
> (despite the hard work of our editors) still have a massive backlog of ugly
> issues like shared memory and other things that prevent implementation in
> JS or similar languages.
>
> My most sincere hope is something good has come out of all this in the
> form of us learning to stay away from prematurely optimized kitchen sink
> APIs and start with the basics in the future.
>
> All the best,
> Jussi
>
>
> On Sat, Oct 19, 2013 at 8:14 PM, s p <sebpiq@gmail.com> wrote:
>
>> > To the extent that it is a problem today, it's partly because
>> present-day implementations are running the JS in these nodes in the main
>> thread.
>>
>> Let's suppose ScriptProcessorNode is optimized and runs in the audio
>> thread? (Therefore minimizing IPC). And let's suppose 2 benchmarks, which
>> for me summarize the important questions.
>>
>> 1)
>> Test1.1 is N ScriptProcessorNodes, each of them running an algorithm A.
>> Test1.2 is one ScriptProcessorNode running N times the algorithm A.
>>
>> 2)
>> You have a really big graph. Test2.1 connects together native AudioNodes
>> and/or ScriptProcessorNodes. Test2.2 implements the exact same dsp graph as
>> a highly optimized dsp function using asm.js, running in a single
>> ScriptProcessorNode.
>>
>> In 1) do you think it is possible to bring the execution time of Test1
>> close to the execution time of Test2 by improving ScriptProcessorNode?
>> In 2) do you think the Test2.1 will always be faster than Test2.2?
>>
>> In fact ... the Test2 could already be done! I should try ...
>>
>>
>>
>> 2013/10/19 Joseph Berkovitz <joe@noteflight.com>
>>
>>>
>>> On Oct 19, 2013, at 12:02 PM, s p <sebpiq@gmail.com> wrote:
>>>
>>>
>>> And no matter if there is more nodes in the futures, there is just no
>>> way all the basic building blocks for all the algorithms humans can ever
>>> conceive can be provided as AudioNodes (and that sucks. Because on
>>> basically every other plateform, there is no limitation).
>>>
>>>
>>> Of course AudioNodes can't be provided for everything. That is why
>>> extensibility is important, and ScriptProcessorNode is at present the
>>> vehicle for doing so.
>>>
>>> Second, if you understand that professionals need things that can't be
>>> built with basic AudioNodes, you understand that ScriptProcessorNode will
>>> be more than just an escape valve.
>>>
>>>
>>> "Escape valve" was an understatement on my part. I completely agree that
>>> ScriptProcessorNode is essential to any professional, wide-ranging use of
>>> the API.
>>>
>>> Now the big problem with that is : you will need to instantiate multiple
>>> ScriptProcessorNodes in your graph, connect them with native AudioNodes,
>>> and because of the sum of the overheads of using ScriptProcessorNodes, you
>>> will end-up in a situation where it is actually more performant to just put
>>> the whole dsp function into ONE single ScriptProcessorNode, re-implementing
>>> oscillators, convolutions, and the whole thing ... making native AudioNodes
>>> useless. That's what I mean by "this architecture is impossible to extend".
>>>
>>>
>>> I don't think your analysis is correct about ScriptProcessorNodes *for
>>> all time*. To the extent that it is a problem today, it's partly because
>>> present-day implementations are running the JS in these nodes in the main
>>> thread. This can impose inter-thread communication overhead that is highly
>>> implementation-dependent. To address this issue does not (to my mind) mean
>>> changing the entire direction of the Web Audio API. It means the overhead
>>> of ScriptProcessorNodes -- or whatever succeeds them in later API versions
>>> -- must be minimized through various means.
>>>
>>> The WG has received similar feedback regarding ScriptProcessorNodes from
>>> other parties as well including internal W3C reviewers. These reviewers
>>> have not concluded that AudioNodes are "useless"; rather, they have
>>> requested that Web Audio address its present shortcomings and made some
>>> positive proposals on how to do so.
>>>
>>>         .            .       .    .  . ...Joe
>>>
>>> *Joe Berkovitz*
>>> President
>>>
>>> *Noteflight LLC*
>>> Boston, Mass. phone: +1 978 314 6271
>>>       www.noteflight.com
>>> "Your music, everywhere"
>>>
>>>
>>
>>
>> --
>> *Sébastien Piquemal
>> *
>> *
>> ** *-----* @sebpiq*
>>   -----* *http://github.com/sebpiq*
>> *
>>  ----- http://funktion.fm
>>
>
>
>


-- 
*Sébastien Piquemal
*
*
** *-----* @sebpiq*
 -----* *http://github.com/sebpiq*
*
 ----- http://funktion.fm