Re: Questioning the current direction of the Web Audio API

2013-10-25 09:04, Jussi Kalliokoski skrev:
> On Mon, Oct 21, 2013 at 9:54 AM, Robert O'Callahan 
> <robert@ocallahan.org <mailto:robert@ocallahan.org>> wrote:
>
>     [Let me note for historical purposes that there was a window of
>     opportunity to promote a more JS-aligned solution over Web Audio,
>     but we missed that window. That was partly my fault for not
>     getting an implementation of MediaStream Processing into Firefox
>     early enough, but I also had practically no support in this group
>     helping me push back against the Webkit "this is what we're
>     shipping" juggernaut.]
>
>
> Practically yes, although I feel that I tried to push for it to the 
> point of annoying the bugger out of the group. I'm not quite sure what 
> was so awesome about this API that made it worth fighting for so 
> fiercely. But that ship sailed, although I did try to pull MSP up even 
> after we had decided to bury it, again not to a very warm welcome.
>
>     It's important to keep in mind when comparing Web APIs to your
>     favourite native audio framework that on the Web, we have to
>     handle untrusted (possibly malicious, very likely poorly written)
>     JS code. That's one reason why running user JS audio processing
>     code on a realtime audio processing thread is a rather trickier
>     proposition for us than for your native audio framework. (And
>     please don't play the "but MY code is perfect" card, it's not
>     helpful.)
>
>
> Native or not, letting people do anything on a high priority thread 
> has its risks. Not as high risks as letting people run arbitrary code 
> on the GPU, but risky anyway, e.g. you can pretty easily brick (until 
> battery removal) a single core device by just having a convolution 
> node with massive uneven kernels.

Yes, you can do pretty evil things with several Web technologies today. 
GLSL is one example (I can make the complete UI crash on my Android 
phone, and I can make a desktop environment completely unresponsive). 
Workers is another example (setting up a dozen of busy workers can 
silently drain the battery of a device). I don't see a big difference in 
doing a heavy convolution in a native audio node or a heavy for-loop in 
a script processor node - both have similar implications, and similar 
solutions (validation, watchdog timers or similar).

>
> Actually security is an argument in favor of a minimal API, because 
> the less we specify, less attack vectors we have likely created. This 
> is because unfortunately spec writers don't write any less imperfect 
> stuff than anyone else.
>
>     So where can we go from here? I think we pretty clearly need a
>     version of ScriptProcessorNode that does JS audio processing off
>     the main thread. The obvious and expedient approach would be to
>     have it dispatch messages to a Worker just like MSP did. It should
>     work with Float32 typed arrays rather than AudioBuffers and be
>     designed to avoid requiring buffer copies. I think that we can
>     arrange for very-short-running JS audio processing to run
>     synchronously during Web Audio processing (by having that realtime
>     thread wait for the JS processing to complete, with a short
>     timeout in case it's slow). We need to define what happens if the
>     processing is too slow; ideally that should just cause a latency
>     increase, if we're not in a true overload situation.
>
>     That should help a lot for a lot of use-cases. If it turns out to
>     be an incomplete solution, we'll have a lot more data to inform
>     our next move.
>
>
> I highly agree with this. In my opinion, this is the sane approach of 
> solving problems little by little, gathering data of the real pain 
> points as we go, and pivoting as needed.
>
> I want to clarify to everyone that I'm not saying that "no native 
> nodes, ever". In fact I'm most of the time an obsessed 
> performance-junkie, which has backlashed at me oh-so-many times, but 
> at least so far my wounds indicate the same things they always say in 
> every performance 101: Don't Optimize Prematurely.
>
> What I'm saying is that native nodes should've come not in v1 or 
> possibly even v2, but once we had perfected the custom processing. 
> Only then we could've started to see the real patterns out in the wild 
> and started to optimize those. Now we just defined arbitrary "basic" 
> building blocks (one could argue against for example convolution being 
> one of them, given that even popular DAWs are lacking 
> implementations), imagined what the use cases for those would be and 
> optimized around them. Premature optimization (and premature 
> abstraction, which is even worse) at its best, in my opinion.
>
> Sorry for throwing more fuel at the fire, but I think it's worth 
> having a little bit of retrospective about things like these every 
> once in a while. :)

Be as it may, I think that debating the merits of different 
would-have-been-solutions is kind of pointless right now, but of course 
let's learn from any mistakes that we may have made in the past and try 
not to repeat them in the future.

In my opinion we should consider the current API feature set 
more-or-less complete for v1 (modulo spec clarifications etc), and move 
on with the future directions of the API. E.g. let's not add more native 
nodes to the API until we have the generic processing path fixed and 
cleared up.

As Olivier pointed out earlier, it might be a good idea to start 
discussing requirements and possible solutions for an 
off-the-main-thread script processor node ASAP. It will quite possibly 
be a difficult task that will need several iterations before we have a 
solution that everyone can be satisfied with.

/Marcus


>
> Cheers,
> Jussi
>
>     Rob
>     -- 
>     Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus eanuttehrotraiitny 
>     eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei
>     csha iids teoa stiheer :p atroa lsyazye,d 'mYaonu,r  "sGients 
>     uapr,e tfaokreg iyvoeunr, 'm aotr  atnod sgaoy ,h o'mGee.t"  uTph
>     eann dt hwea lmka'n?  gBoutt  uIp waanndt  wyeonut  thoo mken.o w *
>     *
>
>


-- 
Marcus Geelnard
Technical Lead, Mobile Infrastructure
Opera Software