Re: ScriptProcessorNode off the main thread (Was: Questioning the current direction of the Web Audio API)

Hi,

Thank you for taking on this issue.
I wish I could help make it happen faster, but I am a bit out of my comfort
zone with all those low-level things (and I apologize in advance if I say
stupid stuff).

> * How well would the one-woker-per-node scale for many nodes (Nx script
contexts)?

Of course this is the first architecture that comes to mind, because the
simplest ... but it sounds a bit scary as in my experience, you can very
quickly end-up with a graph containing a couple hundred nodes, if you are
doing something a bit complex.
So ... 100+ workers? Isn't that a bit insane?

On the other hand, is a solution based on batch processing of several
ScriptProcessorNodes into one or several web worker possible?

> * How about audio params, channel and input/output count control etc:
Should they be in-scope or can we take that later?

I think it is more important to try to get the architecture right. If you
really want AudioParams, you can fake it by calculating ramps and all that
in your node. It wouldn't add any functionality, just an easier way to do
stuff.

Also Jussi told me (I didn't check myself) that if you have ScriptNodes
with different buffer sizes chained together, the latency adds up. If that
is true, I think this is also a big problem and should be addressed.



2013/10/25 Marcus Geelnard <mage@opera.com>

> 2013-10-25 11:09, Olivier Thereaux skrev:
>
>  On 25 Oct 2013, at 08:04, Jussi Kalliokoski <jussi.kalliokoski@gmail.com>
>> wrote:
>>
>>  So where can we go from here? I think we pretty clearly need a version
>>>> of ScriptProcessorNode that does JS audio processing off the main thread.
>>>> The obvious and expedient approach would be to have it dispatch messages to
>>>> a Worker just like MSP did. It should work with Float32 typed arrays rather
>>>> than AudioBuffers and be designed to avoid requiring buffer copies. I think
>>>> that we can arrange for very-short-running JS audio processing to run
>>>> synchronously during Web Audio processing (by having that realtime thread
>>>> wait for the JS processing to complete, with a short timeout in case it's
>>>> slow). We need to define what happens if the processing is too slow;
>>>> ideally that should just cause a latency increase, if we're not in a true
>>>> overload situation.
>>>>
>>>> That should help a lot for a lot of use-cases. If it turns out to be an
>>>> incomplete solution, we'll have a lot more data to inform our next move.
>>>>
>>>>  I highly agree with this. In my opinion, this is the sane approach of
>>> solving problems little by little, gathering data of the real pain points
>>> as we go, and pivoting as needed.
>>>
>> +1
>>
>> We've spent a lot of time in the past discussing worker-only versus
>> main-thread-only and not as much trying to figure out how to make it work.
>> We did, however, get to a resolution to enable ScriptProcessorNode with
>> workers:
>>
>> https://github.com/WebAudio/**web-audio-api/issues/113?**
>> source=cc#issuecomment-**24244840<https://github.com/WebAudio/web-audio-api/issues/113?source=cc#issuecomment-24244840>
>>
>> Anyone has a clear picture of whatever is blocking the inclusion of this
>> resolution into the spec? Or is it just a byproduct of our editorial
>> transition over spring/summer?
>>
>
> While the MSP solution is a possible path forward, I think that we should
> start with agreeing on a few requirements on the new script processor node
> before we can agree that it's the solution that we want.
>
> I think that there are a few differences in the basic ideas of how how MSP
> was envisioned to be used and how Web Audio is designed. Also, we have more
> experience now wrt actual use cases and feature requests.
>
> A few things that come to mind:
>
> * How well would the one-woker-per-node scale for many nodes (Nx script
> contexts)?
>
> * Is the one-worker-per-node design problematic for short-lived nodes
> (worker setup latency etc)?
>
> * From a client architecture point of view:
>   - Can we support running these workers in high-priority threads to
> minimize latency?
>   - Should the audio workers have special GC considerations in order to
> minimize worst-case latency?
>
> * As a consequence of the above points (and possibly others): Should the
> "workers" that we use be regular Web workers, dumbed down Web workers, or
> something else?
>
> * How about audio params, channel and input/output count control etc:
> Should they be in-scope or can we take that later?
>
>
>
>> I would like to see a group of people to come up with a detailed proposal
>> if there are remaining hurdles stopping us from including this in the spec,
>> and proposing a spec patch complete with interfaces based on MSP.
>>
>> Also, may I suggest ROC should lead this? What say you, ROC?
>>
>
> I'd like to see ROC head this group too.
>
> /Marcus
>
>
>
>> Thanks,
>> --
>> Olivier
>>
>>
>>
>>
>>
>> -----------------------------
>> http://www.bbc.co.uk
>> This e-mail (and any attachments) is confidential and
>> may contain personal views which are not the views of the BBC unless
>> specifically stated.
>> If you have received it in
>> error, please delete it from your system.
>> Do not use, copy or disclose the
>> information in any way nor act in reliance on it and notify the sender
>> immediately.
>> Please note that the BBC monitors e-mails
>> sent or received.
>> Further communication will signify your consent to
>> this.
>> -----------------------------
>>
>>
>
> --
> Marcus Geelnard
> Technical Lead, Mobile Infrastructure
> Opera Software
>
>


-- 
*Sébastien Piquemal
*
*
** *-----* @sebpiq*
 -----* *http://github.com/sebpiq*
*
 ----- http://funktion.fm