Re: AudioWorker proposal

Chris,

Thanks so much for clarifying.

I think we’re on the same page here. The only additional suggestion I’m making (recapping from the beginning of this thread) is that if we don’t use AudioProcessingEvent (and I now think it would be clearest *not* to, so there is no confusion about the difference in mechanisms) let’s use some other single object type to collect the information passed into and out of the AudioWorker so that we do not have a function signature that needs to evolve over time. In keeping with what else you said, perhaps let’s not call that type an Event either, since that confuses things and suggests asynchronicity (and furthermore implies onXXX() and addEventListener(“XXX”,…)). This new type will be an object that communicates information between the audio thread and an AudioWorker, period.

How does that sound to you and other folks on this thread?


.            .       .    .  . ...Joe

Joe Berkovitz
President

Noteflight LLC
Boston, Mass.
phone: +1 978 314 6271
www.noteflight.com
"Your music, everywhere"



On Jul 21, 2014, at 5:08 PM, Chris Wilson <cwilso@google.com> wrote:

> Hey Joe-
> 
> What I'd been intending was that the AudioProcessingEvent (and associated audioprocess event) would go away, because events are typically asynchronous - i.e. "audioprocess" is essentially a function that's called to process audio, much as the onaudioprocess event handler used to be called in standard scenarios, but it's called directly from the audio process when it needs data.  
> 
> Upon further reflection, perhaps we don't need to be that independent, as it's the firing of events that is typically asynchronous (via PostMessage).  The primary freedom I want to make sure we're obtaining is that the audio processing function can be called synchronously, and we should NOT need to create a new set of i/o objects for every call (to eliminate/reduce garbage collection in this thread).  I was always on the fence about using an Event here anyway, because I don't think multiple handlers is a good pattern, but I'm somewhat ambivalent and open to influence.  Please note, though, that this event is coming from the audio engine, and is delivered to the Worker; you wouldn't be able to use Event to pass parameters from the main JS thread to the worker thread; you'll still need to use postMessage for setting up and varying parameters, etc.
> 
> So: either we use the synchronous callback mechanism, as detailed previously in this thread modulo the error I made in not separating input and output buffers :):
> 
> [Constructor(DOMString scriptURL, optional unsigned long bufferSize)]
> interface AudioWorker : Worker {
> };
> 
> callback AudioProcessCallback = void (double playbackTime, AudioBuffer inputBuffer, AudioBuffer outputBuffer);
> 
> interface AudioWorkerGlobalScope : DedicatedWorkerGlobalScope {
>   attribute AudioProcessCallback audioprocess;
>   readonly attribute float sampleRate;
> };
> 
> interface ScriptProcessorNode : AudioNode {
>   readonly attribute double latency;
>   readonly attribute AudioWorker worker;
> };
> 
> partial interface AudioContext {
>   ScriptProcessorNode createScriptProcessor(
>     DOMString scriptURL,
>     optional unsigned long bufferSize = 0,
>     optional unsigned long numberOfInputChannels = 2,
>     optional unsigned long numberOfOutputChannels = 2);
> }
> 
> 
> , or we keep the event-based system (although as the event is firing from the audio system inside the same thread, it would in practice be called directly):
> [Constructor(DOMString scriptURL, optional unsigned long bufferSize)]
> interface AudioWorker : Worker {
> };
> 
> interface AudioWorkerGlobalScope : DedicatedWorkerGlobalScope {
>   attribute EventHandler onaudioprocess;
>   readonly attribute float sampleRate;
> };
> 
> interface ScriptProcessorNode : AudioNode {
>   readonly attribute double latency;
>   readonly attribute AudioWorker worker;
> };
> 
> interface AudioProcessingEvent : Event {
>     readonly    attribute double      playbackTime;
>     readonly    attribute AudioBuffer inputBuffer;
>     readonly    attribute AudioBuffer outputBuffer;
> };
> 
> partial interface AudioContext {
>   ScriptProcessorNode createScriptProcessor(
>     DOMString scriptURL,
>     optional unsigned long bufferSize = 0,
>     optional unsigned long numberOfInputChannels = 2,
>     optional unsigned long numberOfOutputChannels = 2);
> }
> 
> 
> On Mon, Jul 21, 2014 at 1:00 PM, Joseph Berkovitz <joe@noteflight.com> wrote:
> Chris,
> 
> I understand that postMessage and onmessage are available to any Worker subclass. What I am looking for is an explanation of the AudioProcessCallback type and “audioprocess” attribute in the global namespace, cited in the IDL that you posted in issue 113 on March 26.
> 
> If you’re just saying we’re going to be using onaudioprocess and AudioProcessingEvent for these new-style workers, then I don’t see how the proposed IDL fits into that. And if it’s obsolete, that’s OK.
> 
> …Joe
> 
> 
> On Jul 18, 2014, at 3:27 PM, Chris Wilson <cwilso@google.com> wrote:
> 
>> No, since AudioWorker is derived from Worker, it responds to (and has access to) .postMessage and .onmessage - enabling you to send and receive messages between the main thread and the audio worker, even structured objects (if they're clonable). (See https://developer.mozilla.org/en-US/docs/Web/API/Worker.postMessage.)  The isolation of everything else is a feature.  You could, for example, transfer an ArrayBuffer, so you could implement convolution or something else that requires a data array - but you don't need to worry about race conditions, because it's message-based and transferable or clonable only.
>> 
>> In this way, we can also not worry about the neutering/reuse/new'ing of ArrayBuffers for the onaudioprocess calls - because they're not across threads - and this is a desirable optimization, since we don't need to thrash memory (by alloc'ing new objects for every audioprocess call).
>> 
>> 
>> 
>> 
>> On Fri, Jul 18, 2014 at 12:05 PM, Joseph Berkovitz <joe@noteflight.com> wrote:
>> Thanks, Chris. I think you are saying the worker script will effectively do something like this:
>> 
>> // Assign global audio processing callback for this worker
>> audioprocess = function(playbackTime, buffer) {
>>   //…handle input passed via the buffer parameter
>> };
>> 
>> The one concern I see with this is that it may make it harder to pass a flexible bunch of information to the worker — this information is currently burned into the AudioProcessCallback parameter list as a playback time and a buffer. Even if we don’t use an AudioProcessingEvent, wouldn’t some kind of single object (AudioProcessingData?) holding the worker’s input and output data be easier to evolve over time than a parameter list, if there are changes or alternatives down the road?
>> 
>> Also, I’m wondering why the sampleRate is provided in the global scope and not as an parameter to the callback (or, if you accept my idea, an attribute of some AudioProcessingData object)?
>> 
>> …Joe
>> 
>> On Jul 18, 2014, at 12:57 PM, Chris Wilson <cwilso@google.com> wrote:
>> 
>>> Yes, that's somewhat correct.  The only bit that isn't correct is that it probably shouldn't really be an AudioProcessingEvent - because since the goal is to be synchronous, it should be called directly, not posted or fired.  Something like (totally off the top of my head):
>>> 
>>> [Constructor(DOMString scriptURL, optional unsigned long bufferSize)]
>>> interface AudioWorker : Worker {
>>> };
>>> 
>>> callback AudioProcessCallback = void (double playbackTime, AudioBuffer buffer);
>>> interface AudioWorkerGlobalScope : DedicatedWorkerGlobalScope {
>>>   attribute AudioProcessCallback audioprocess;
>>>   readonly attribute float sampleRate;
>>> };
>>> 
>>> interface ScriptProcessorNode : AudioNode {
>>>   readonly attribute double latency;
>>>   readonly attribute AudioWorker worker;
>>> };
>>> 
>>> partial interface AudioContext {
>>>   ScriptProcessorNode createScriptProcessor(
>>>     DOMString scriptURL,
>>>     optional unsigned long bufferSize = 0,
>>>     optional unsigned long numberOfInputChannels = 2,
>>>     optional unsigned long numberOfOutputChannels = 2);
>>> }
>>> 
>>> 
>>> On Thu, Jul 17, 2014 at 11:28 AM, Joseph Berkovitz <joe@noteflight.com> wrote:
>>> Hi Chris,
>>> 
>>> I remember the issue (and contributed some comments to it at the time) but it’s difficult to determine the exact proposal that was discussed at last week’s meeting, since a number of variants are proposed in github issue 113 with different folks suggesting this flavor or that flavor.
>>> 
>>> From the fact that Olivier called this “synchronous”, I'm guessing that you were discussing an AudioWorker object that executes directly in the audio thread and doesn’t have to deal with message-passing. I infer that an AudioWorker obtains its input (and output?) audio buffer directly from an AudioProcessingEvent passed to the worker’s “onaudioprocess” function defined in its AudioWorkerGlobalScope. This was proposed in your comment of March 26, I think.
>>> 
>>> Is that understanding correct? Were further changes to this approach discussed?
>>> 
>>> …Joe
>>> 
>>> On Jul 17, 2014, at 1:03 PM, Chris Wilson <cwilso@google.com> wrote:
>>> 
>>>> It's mostly written up in the issue (https://github.com/WebAudio/web-audio-api/issues/113), but I realized on the call that I hadn't yet written out an IDL for how the synchronous version would work.
>>>> 
>>>> 
>>>> On Thu, Jul 17, 2014 at 9:53 AM, Paul Adenot <paul@paul.cx> wrote:
>>>> On Thu, Jul 17, 2014, at 06:43 PM, Joseph Berkovitz wrote:
>>>>> Hi Olivier,
>>>>>  
>>>>> On Jul 11, 2014, at 6:16 AM, Olivier Thereaux <olivier.thereaux@bbc.co.uk> wrote:
>>>>>>  
>>>>>> * Progress of scriptprocessornodes in workers issue
>>>>>> We discussed a proposal for a synchronous model for the node. There will also be a Call for Consensus soon on the possibility of breaking changes for the main thread ScriptProcessorNode.
>>>>>>  
>>>>>  
>>>>> Is there a writeup of the proposal that was discussed?
>>>>  
>>>> Not as far as I know. We kind of agreed on a super high level, but nothing very solid yet.
>>>>  
>>>> Paul.
>>>> 
>>> 
>>> 
>>> 
>>> .            .       .    .  . ...Joe
>>> 
>>> Joe Berkovitz
>>> President
>>> 
>>> Noteflight LLC
>>> Boston, Mass. phone: +1 978 314 6271
>>> www.noteflight.com
>>> "Your music, everywhere"
>>> 
>>> 
>> 
>> 
>> 
> 
> .            .       .    .  . ...Joe
> 
> Joe Berkovitz
> President
> 
> Noteflight LLC
> Boston, Mass. phone: +1 978 314 6271
> www.noteflight.com
> "Your music, everywhere"
> 
>