Re: ScriptProcessorNode off the main thread (Was: Questioning the current direction of the Web Audio API)

For v2, the opposite direction to putting script nodes in workers has some potential as well, I think.

The intent of script nodes is to make custom processing possible. It is only incidental that the processing is itself specified in JS. For instance, if today JS were only good enough as a glue language (ex: 50x slower than C), we won't even be talking about such JS script nodes.

So by the "opposite direction", I mean the question of "what does it take to put JS code in the same high priority audio thread as the native nodes?". Before shooting this down, let's look at the benefits -

1. Familiar language that's close to C in looks as well as performance.
2. Realtime JS code can run in sync with the other native nodes.
3. A JS node will (finally) be able to emulate any native node without introducing additional delays.
4. The GC behaviour (formerly "dynamic lifetime") of a JS node can also be identical to the native nodes.
5. All JS nodes attached to a context can be run in the same thread.

This is all good, but the factors that make it hard to just say "yup let's do this" are -

a. GC in an RT thread is bad bad bad.
b. DOM interaction will be super complicated and also bad bad bad in the RT thread.
c. There is no "c." as far as I can tell (fingers crossed).

So what if with each AudioContext thread, we also instantiate a JS environment, forbid it to do GC and forbid it to interact with the DOM? Would we then have a workable solution? At least this would be as flexible as putting the script into workers as those come with the DOM limitation anyway. Well, maybe the allocation may limit it a bit more, but not by much I think.

What does it take to prevent GC? Here is a possible definition -

1. The onaudioprocess function has to be a pure function of the node's inputs and outputs and properties. i.e. it cannot close over objects in the main thread's environment. In other words, the onaudioprocess function should be representable entirely by its source code.

2. The function must never allocate objects. This forbids "new", "{}", "[]", string concatenation and also changes to the sizes of arrays and objects. For DSP code, this doesn't seem draconian to me.

If I understand it right, the asm.js spec pretty much fits this bill. If the onaudioprocess function has a spec something like - function (API, params[], inputs[], outputs[]) { /* ... */ return sampleFramesGenerated; } - where API provides some standard utility functions including Math, DSP, etc., params[] are AudioParams' values as Float32Arrays and inputs[] and outputs[] are Float32Arrays corresponding to the node's pins.

What about libraries? Source code composition is one way to provide libraries, similar to the way shaders are composed in the 3D graphics universe. Maybe some ideas about shader linking can be borrowed into this spec as well .. perhaps in v3. For even source code composition though, we'd need nested function support, but we won't need closure support.

I'd guess that the computer music community will love this!

-Kumar

On 25 Oct, 2013, at 3:06 PM, Marcus Geelnard <mage@opera.com> wrote:

> 2013-10-25 11:09, Olivier Thereaux skrev:
>> On 25 Oct 2013, at 08:04, Jussi Kalliokoski <jussi.kalliokoski@gmail.com> wrote:
>> 
>>>> So where can we go from here? I think we pretty clearly need a version of ScriptProcessorNode that does JS audio processing off the main thread. The obvious and expedient approach would be to have it dispatch messages to a Worker just like MSP did. It should work with Float32 typed arrays rather than AudioBuffers and be designed to avoid requiring buffer copies. I think that we can arrange for very-short-running JS audio processing to run synchronously during Web Audio processing (by having that realtime thread wait for the JS processing to complete, with a short timeout in case it's slow). We need to define what happens if the processing is too slow; ideally that should just cause a latency increase, if we're not in a true overload situation.
>>>> 
>>>> That should help a lot for a lot of use-cases. If it turns out to be an incomplete solution, we'll have a lot more data to inform our next move.
>>>> 
>>> I highly agree with this. In my opinion, this is the sane approach of solving problems little by little, gathering data of the real pain points as we go, and pivoting as needed.
>> +1
>> 
>> We've spent a lot of time in the past discussing worker-only versus main-thread-only and not as much trying to figure out how to make it work. We did, however, get to a resolution to enable ScriptProcessorNode with workers:
>> 
>> https://github.com/WebAudio/web-audio-api/issues/113?source=cc#issuecomment-24244840
>> 
>> Anyone has a clear picture of whatever is blocking the inclusion of this resolution into the spec? Or is it just a byproduct of our editorial transition over spring/summer?
> 
> While the MSP solution is a possible path forward, I think that we should start with agreeing on a few requirements on the new script processor node before we can agree that it's the solution that we want.
> 
> I think that there are a few differences in the basic ideas of how how MSP was envisioned to be used and how Web Audio is designed. Also, we have more experience now wrt actual use cases and feature requests.
> 
> A few things that come to mind:
> 
> * How well would the one-woker-per-node scale for many nodes (Nx script contexts)?
> 
> * Is the one-worker-per-node design problematic for short-lived nodes (worker setup latency etc)?
> 
> * From a client architecture point of view:
>  - Can we support running these workers in high-priority threads to minimize latency?
>  - Should the audio workers have special GC considerations in order to minimize worst-case latency?
> 
> * As a consequence of the above points (and possibly others): Should the "workers" that we use be regular Web workers, dumbed down Web workers, or something else?
> 
> * How about audio params, channel and input/output count control etc: Should they be in-scope or can we take that later?
> 
>> 
>> I would like to see a group of people to come up with a detailed proposal if there are remaining hurdles stopping us from including this in the spec, and proposing a spec patch complete with interfaces based on MSP.
>> 
>> Also, may I suggest ROC should lead this? What say you, ROC?
> 
> I'd like to see ROC head this group too.
> 
> /Marcus
> 
>> 
>> Thanks,
>> --
>> Olivier
>> 
>> 
>> 
>> 
>> 
>> -----------------------------
>> http://www.bbc.co.uk
>> This e-mail (and any attachments) is confidential and
>> may contain personal views which are not the views of the BBC unless specifically stated.
>> If you have received it in
>> error, please delete it from your system.
>> Do not use, copy or disclose the
>> information in any way nor act in reliance on it and notify the sender
>> immediately.
>> Please note that the BBC monitors e-mails
>> sent or received.
>> Further communication will signify your consent to
>> this.
>> -----------------------------
>> 
> 
> 
> -- 
> Marcus Geelnard
> Technical Lead, Mobile Infrastructure
> Opera Software
>