Re: Sync of scriptProcessorNode and native Node

Thank you for your answers, Rusell! I comment some things between lines:

>> So, the audio system has determined that it needs a buffer from you
>> that will be played at a specific time, and it is your job to fill up
>> that buffer's audio. When you return from the event callback, that
>> audio will be transferred to the audio system.  The audio system has
>> already determined when the buffer you are working on should be
>> played, and that's the playbackTime timestamp.  If you return from the
>> event callback in time for the audio system to play back that buffer
>> before playbackTime, the buffer will be played at playbackTime.  If
>> you don't return from the callback in time, who knows what will
>> happen?  (I'm not sure if this is even specified).

As I understand, if you don't return from the callback in time, there are some kind of audio breakups or glitches. Therefore, if you have glitches, you have to increase the size of your scriptProcessorNode bufferSize.

>> As far as how playbackTime is calculated, that's not really your
>> concern, as you're not the one implementing the audio system.  It
>> should be fairly complicated.  If everything is going smoothly though,
>> it should at least increment by one buffer every callback - otherwise,
>> something is dropping audio somewhere.  It has no bearing at all on
>> the time your audio callback is called (is this what you mean by
>> "processTime"?)- it only refers to the time that the buffer will be
>> played.

I know that I don't need to know how it is exactly calculated to work with it, but I would like to know it, for my own knowledge! :-)
I found the discussions of  implementing playbackTime in chrome: https://codereview.chromium.org/210973002
Currently, in Chromium is implemented as:
playbackTime = (currentSampleFrame + bufferSize)/ sampleRate
where currentSampleFrame counts the number of sample-frames processed by the destination.


Thanks,

Arnau



On 17 mai 2014, at 03:41, "Srikumar K. S." <srikumarks@gmail.com> wrote:

>> Chris - Yes, windows (at least the newer WASAPI) reports the stream
>> latency (a function called "GetStreamLatency" on the audio client).
> 
> The thread [1] mentions IAudioClock::GetPosition as possibly a more
> accurate way to get to the latency info. However, the description in the 
> MSDN man page [2] is not as direct as the description in the thread.
> 
> [1] http://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/c4fd959b-a0c8-40a0-b037-f949c2428337/audio-inputoutput-synchronization-under-wasapi
>   "IAudioClock::GetPosition will tell you the sample that is 
>    hitting the air at the speaker cone (for render) or is coming in from 
>    the air (for the microphone.)"
> 
> [2] http://msdn.microsoft.com/en-us/library/windows/desktop/dd370889(v=vs.85).aspx
> 
> -Kumar
> 
> On 17 May, 2014, at 6:37 am, Russell McClellan <russell.mcclellan@gmail.com> wrote:
> 
>> Chris - Yes, windows (at least the newer WASAPI) reports the stream
>> latency (a function called "GetStreamLatency" on the audio client).
>> Linux is trickier and I'm not sure about it.  All professional DAWs
>> for mac and windows display the output latency of the interface.
>> 
>> Also, I've said this before, but as a web developer, script processors
>> in workers would be a total godsend.  It would make my life a *lot*
>> easier.
>> 
>> Arnau -
>> 
>> First of all, I think your test should work, and the bug is in the
>> browser (although, it would be better if you had set the gain to 4 in
>> the second stage, so that a correct browser would play an unbroken
>> sine wave).  As I mentioned previously, I have never had any success
>> with playbackTime in firefox.  In chrome, it at least used to work, I
>> believe.
>> 
>> There's very little high-level documentation here, so I think you may
>> be a bit confused about the general architecture of the web audio api.
>> When you get an "audio processing event", you are in the main
>> javascript thread.  The audio processing callback is called using
>> literally the same mechanism as any other event - clicks, or whatever.
>> There's nothing concurrent or different about audio processing
>> callbacks than any other sort of event handler.  That's why the global
>> variable update works.  It's just like if a user was clicking on a
>> button 100 times a second or so.  You could update a global variable
>> on a timer, but while you're in the click handler, you wouldn't see
>> the global variable update until you returned and were called again.
>> 
>> So, the audio system has determined that it needs a buffer from you
>> that will be played at a specific time, and it is your job to fill up
>> that buffer's audio. When you return from the event callback, that
>> audio will be transferred to the audio system.  The audio system has
>> already determined when the buffer you are working on should be
>> played, and that's the playbackTime timestamp.  If you return from the
>> event callback in time for the audio system to play back that buffer
>> before playbackTime, the buffer will be played at playbackTime.  If
>> you don't return from the callback in time, who knows what will
>> happen?  (I'm not sure if this is even specified).
>> 
>> As far as how playbackTime is calculated, that's not really your
>> concern, as you're not the one implementing the audio system.  It
>> should be fairly complicated.  If everything is going smoothly though,
>> it should at least increment by one buffer every callback - otherwise,
>> something is dropping audio somewhere.  It has no bearing at all on
>> the time your audio callback is called (is this what you mean by
>> "processTime"?)- it only refers to the time that the buffer will be
>> played.
>> 
>> Thanks,
>> -Russell
>> 
>> 
>> 
>> 
>> On Thu, May 8, 2014 at 4:09 PM, Chris Wilson <cwilso@google.com> wrote:
>>> Hmm, interesting.  Any Windows/Linux sound api consumers want to contribute
>>> here?  (I couldn't find a comparable property in Windows APIs with a quick
>>> look.)
>>> 
>>> 
>>> On Wed, May 7, 2014 at 9:58 PM, Srikumar K. S. <srikumarks@gmail.com> wrote:
>>>> 
>>>> (e.g., your Bluetooth example - I'm not sure there's a way to detect that
>>>> latency!)
>>>> 
>>>> 
>>>> There is ... at least in iOS and MacOSX. I use it in my iOS app. When the
>>>> audio route changes, I just ask for the "CurrentHardwareOutputLatency"
>>>> property
>>>> of the AudioSession.
>>>> 
>>>> Even if the internal buffering is the only latency that the system can
>>>> access,
>>>> that would still be useful to have explicitly via the API than not have
>>>> it.
>>>> This would permit the API implementations to account for such latency
>>>> info where and when it is available.
>>>> 
>>>> -Kumar
>>>> 
>>>> On 7 May, 2014, at 11:43 pm, Chris Wilson <cwilso@google.com> wrote:
>>>> 
>>>> Although this is definitely still an issue (Issue #12, as a matter of
>>>> fact!  https://github.com/WebAudio/web-audio-api/issues/12), I would like to
>>>> caution that we cannot necessarily fix this entirely.  IIRC, in a number of
>>>> cases, we simply do not know what latency is caused by the hardware device
>>>> itself; I think we can only account for the buffering latency in our own
>>>> systems.  (e.g., your Bluetooth example - I'm not sure there's a way to
>>>> detect that latency!)
>>>> 
>>>> 
>>>> On Tue, May 6, 2014 at 9:54 PM, Srikumar K. S. <srikumarks@gmail.com>
>>>> wrote:
>>>>> 
>>>>> There is also a different "sync" issue that is yet to be addressed.
>>>>> Currently, we do not have a way to translate between a time expressed in the
>>>>> AudioContext.currentTime coordinates into the DOMHighResTimeStamp
>>>>> coordinates. Times in requestAnimationFrame are DOMHighResTimeStamp times
>>>>> (iirc) and synchronizing visuals with computed audio is near impossible
>>>>> without a straightforward way to translate between them. This gets worse on
>>>>> mobile devices where a bluetooth speaker can get connected while an audio
>>>>> context is running and on-the-fly add 300ms of latency.
>>>>> 
>>>>> I don't think I've missed any development towards this, but if I have, I
>>>>> apologize for raising this again and am all ears to hear the solution.
>>>>> 
>>>>> -Kumar
>>>>> sriku.org
>>>>> 
>>>>> 
>>>>> On 7 May, 2014, at 12:36 am, Joseph Berkovitz <joe@noteflight.com> wrote:
>>>>> 
>>>>> To echo Chris W, it is *possible* to sync by paying attention to the
>>>>> playbackTime of an audio processing event, and by scheduling parameter
>>>>> changes and actions on native nodes in relation to that time (which is
>>>>> expressed in the AudioContext timebase).
>>>>> 
>>>>> However, due to the fact that the code in a  ScriptProcessorNode runs in
>>>>> the main JS thread, as a practical matter it is difficult to do such syncing
>>>>> reliably and robustly without glitching. There are also some browser
>>>>> portability issues as Chris mentioned.
>>>>> 
>>>>> Hence the urgency to find a better solution.
>>>>> 
>>>>> .            .       .    .  . ...Joe
>>>>> 
>>>>> Joe Berkovitz
>>>>> President
>>>>> 
>>>>> Noteflight LLC
>>>>> Boston, Mass. phone: +1 978 314 6271
>>>>> www.noteflight.com
>>>>> "Your music, everywhere"
>>>>> 
>>>>> On May 6, 2014, at 4:13 AM, Arnau Julia <Arnau.Julia@ircam.fr> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> I'm aware of public-audio list conversations about the use of workers for
>>>>> the scriptProcessorNode and I'm very excited about the possibilities of this
>>>>> solution, but I supposed that it was possible to sync a scriptProcessorNode
>>>>> and a native Node with the current implementation. Am I wrong? And if not,
>>>>> how it is possible to achieve?
>>>>> 
>>>>> Thank you,
>>>>> 
>>>>> Arnau
>>>>> 
>>>>> On 5 mai 2014, at 18:42, Chris Wilson <cwilso@google.com> wrote:
>>>>> 
>>>>> Lonce,
>>>>> 
>>>>> this is one of the biggest and most important issues on my Web Audio
>>>>> plate right now.  I'm working on figuring out how to spark some coming
>>>>> together of implementers over the summer to come up with a workable
>>>>> solution.
>>>>> 
>>>>> 
>>>>> On Fri, May 2, 2014 at 9:38 PM, lonce <lonce.audio@sonic.zwhome.org>
>>>>> wrote:
>>>>>> 
>>>>>> 
>>>>>> Hi -
>>>>>> 
>>>>>>   I think the real question is not how to hack this, but the status of
>>>>>> progress on a fundamental solution to this  Achilles heal of the current
>>>>>> system. From what I gather, the solution will probably be in the form of web
>>>>>> workers (?), but I don't know how much attention this is getting now.
>>>>>>   Once this is solved, the system becomes truly extensible and I am
>>>>>> sure it will open up an explosive era of community development just waiting
>>>>>> to happen!
>>>>>> 
>>>>>> Best,
>>>>>>            - lonce
>>>>>> 
>>>>>> 
>>>>>> On 5/2/2014 4:39 PM, Arnau Julia wrote:
>>>>>>> 
>>>>>>> Hello,
>>>>>>> 
>>>>>>> First of all, thank for all your answers.
>>>>>>> 
>>>>>>>> The first thing to note is that all script processor node processing
>>>>>>>> happens on the main javascript thread.  This means if you change a
>>>>>>>> global variable in another part of your javascript program, it will
>>>>>>>> show definitely show up on the next AudioProcessingEvent.  So, that
>>>>>>>> answers your first problem - once you set the variable in your
>>>>>>>> javascript, on the next buffer the change will be there.  There's no
>>>>>>>> parallelism at all in the javascript - there's only one thing
>>>>>>>> happening at once.
>>>>>>> 
>>>>>>> I would like to understand how it works. The difference that I found
>>>>>>> between the scriptProcessorNode and the 'native' AudioNode Interface is that
>>>>>>> the first uses a Event Handler and the AudioNodes are EventTargets. Is it
>>>>>>> the reason why the global variables are updated only one time for each
>>>>>>> buffer? Someone have more documentation to understand it more deeply?
>>>>>>> 
>>>>>>>> For your second question, you need some sort of timestamp on the
>>>>>>>> buffer.  The web audio api provides this as the playbackTime field on
>>>>>>>> the AudioProcessingEvent.  Of course, you only have access to the
>>>>>>>> playback time of the buffer you are currently processing, but you can
>>>>>>>> guess when the next playbackTime will be by setting the last processed
>>>>>>>> time as a global variable, and then adding one buffer's worth of time
>>>>>>>> to that to get the next playbackTime.  This will be fine unless you
>>>>>>>> drop buffers, in which case you're probably not worried about a smooth
>>>>>>>> ramp :-).  So, one easy solution to your second problem is to always
>>>>>>>> store the last playback time that each of your script nodes processed,
>>>>>>>> and then start the ramp on the *next* buffer.  The spec guarantees
>>>>>>>> that the playbackTime and ramping is sample accurate, so no worries
>>>>>>>> there.  In practice, the last time I checked, which was over a year
>>>>>>>> ago, firefox had serious problems with the playbackTime field (I don't
>>>>>>>> remember if it was just absent or if it had some other problem that
>>>>>>>> made it unusable.)
>>>>>>> 
>>>>>>> It seems a good solution! I didn't found the playbackTime on the last
>>>>>>> stable version of Chrome but I found it in Firefox. Is there any alternative
>>>>>>> for Chrome?
>>>>>>> 
>>>>>>> I have done some basic experiments with playbackTime in Firefox and it
>>>>>>> seems that is not totally sync or maybe I don't understand how to use it. I
>>>>>>> uploaded the experiment to jsfiddle (only Firefox!):
>>>>>>> http://jsfiddle.net/PgeLv/11/
>>>>>>> The experiment structure is:
>>>>>>> oscillatorNode (source) ----> scriptProcesorNode -----> GainNode
>>>>>>> -------> Destination
>>>>>>> 
>>>>>>> On the other hand, I would like to understand 'what' is exactly the
>>>>>>> playbackTime. I guess that it can be something like that:
>>>>>>> 
>>>>>>> playbackTime = bufferSize/sampleRate + 'processTime' + 'wait interval
>>>>>>> until the event return the data to the audio thread'
>>>>>>> 
>>>>>>> If this hypothesis is true, it means that the playbackTime is different
>>>>>>> for each event, because it depends on the activity of the general thread.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Arnau
>>>>>>> 
>>>>>>> On 22 avr. 2014, at 01:51, Russell McClellan
>>>>>>> <russell.mcclellan@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hey Arnau -
>>>>>>>> 
>>>>>>>> Yes, this is probably underdocumented.  The good news is, the
>>>>>>>> designers of the web audio api do actually have an answer for linking
>>>>>>>> native nodes and script processor nodes.
>>>>>>>> 
>>>>>>>> The first thing to note is that all script processor node processing
>>>>>>>> happens on the main javascript thread.  This means if you change a
>>>>>>>> global variable in another part of your javascript program, it will
>>>>>>>> show definitely show up on the next AudioProcessingEvent.  So, that
>>>>>>>> answers your first problem - once you set the variable in your
>>>>>>>> javascript, on the next buffer the change will be there.  There's no
>>>>>>>> parallelism at all in the javascript - there's only one thing
>>>>>>>> happening at once.
>>>>>>>> 
>>>>>>>> For your second question, you need some sort of timestamp on the
>>>>>>>> buffer.  The web audio api provides this as the playbackTime field on
>>>>>>>> the AudioProcessingEvent.  Of course, you only have access to the
>>>>>>>> playback time of the buffer you are currently processing, but you can
>>>>>>>> guess when the next playbackTime will be by setting the last processed
>>>>>>>> time as a global variable, and then adding one buffer's worth of time
>>>>>>>> to that to get the next playbackTime.  This will be fine unless you
>>>>>>>> drop buffers, in which case you're probably not worried about a smooth
>>>>>>>> ramp :-).  So, one easy solution to your second problem is to always
>>>>>>>> store the last playback time that each of your script nodes processed,
>>>>>>>> and then start the ramp on the *next* buffer.  The spec guarantees
>>>>>>>> that the playbackTime and ramping is sample accurate, so no worries
>>>>>>>> there.  In practice, the last time I checked, which was over a year
>>>>>>>> ago, firefox had serious problems with the playbackTime field (I don't
>>>>>>>> remember if it was just absent or if it had some other problem that
>>>>>>>> made it unusable.)
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> -Russell
>>>>>>>> 
>>>>>>>> On Fri, Apr 18, 2014 at 10:50 AM, Casper Schipper
>>>>>>>> <casper.schipper@monotonestudio.nl> wrote:
>>>>>>>>> 
>>>>>>>>> Dear Arnau,
>>>>>>>>> 
>>>>>>>>> this is indeed a frustrating (but probably performance wise
>>>>>>>>> necessary)
>>>>>>>>> limitation of the normal web audio nodes,
>>>>>>>>> parameters in a scriptProcessorNode can only be updated once every
>>>>>>>>> vector
>>>>>>>>> which is a minimum of 256 samples.
>>>>>>>>> 
>>>>>>>>> Maybe you could solve your problem by using one of the javascript
>>>>>>>>> libraries
>>>>>>>>> that  bypass most of web audio api and do everything in JS itself.
>>>>>>>>> What comes first to mind would be the Gibberish.js library by Charlie
>>>>>>>>> Roberts, which gives you the ability to control parameters per sample
>>>>>>>>> and
>>>>>>>>> easily schedule synchronized parameter changes also with sample
>>>>>>>>> accuracy:
>>>>>>>>> http://www.charlie-roberts.com/gibberish/docs.html
>>>>>>>>> It should be quite easy to extend it with your own nodes.
>>>>>>>>> There are other libraries as well like flocking.js and Timbre.js.
>>>>>>>>> 
>>>>>>>>> Of course it comes with some performance penalties, but Gibberish
>>>>>>>>> tries to
>>>>>>>>> at least generate javascript code that should be as efficient as
>>>>>>>>> possible
>>>>>>>>> for it's JIT complication style, as far as it's own nodes are
>>>>>>>>> considered.
>>>>>>>>> 
>>>>>>>>> Hope it helps,
>>>>>>>>> Casper
>>>>>>>>> 
>>>>>>>>> casper.schipper@monotonestudio.nl
>>>>>>>>> Mauritskade 55C (the thinking hut)
>>>>>>>>> 1092 AD  Amsterdam
>>>>>>>>> +316 52 322 590
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 18 apr. 2014, at 10:55, Arnau Julia <Arnau.Julia@ircam.fr> wrote:
>>>>>>>>> 
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>> I'm trying to synchronizing the buffer in a scriptProcessorNode with
>>>>>>>>> native/regular web audio nodes and I'm having some problems. My
>>>>>>>>> problem is
>>>>>>>>> that I want to synchronize the scriptProcessorNode with a ramp of a
>>>>>>>>> GainNode.
>>>>>>>>> 
>>>>>>>>> My program looks like the attached diagram. Each scriptProcessorNode
>>>>>>>>> is a
>>>>>>>>> filter with n coefficients and these coefficients are in a global
>>>>>>>>> variable.
>>>>>>>>> My problem comes when I try to update these coefficients and do a
>>>>>>>>> ramp in
>>>>>>>>> the gain through an audioParam at the "same time".
>>>>>>>>> 
>>>>>>>>> The start scenario is (in pseudo-code):
>>>>>>>>> 
>>>>>>>>> audioBufferSourceNode.connect(scriptProcessorNode0);
>>>>>>>>> audioBufferSourceNode.connect(scriptProcessorNode1);
>>>>>>>>> 
>>>>>>>>> scriptProcessorNode0.connect(gainNode0);
>>>>>>>>> scriptProcessorNode0.connect(gainNode1);
>>>>>>>>> 
>>>>>>>>> gainNode0.connect(audioContext.destination);
>>>>>>>>> gainNode1.connect(audioContext.destination);
>>>>>>>>> 
>>>>>>>>> gainNode1.gain.value = 0;
>>>>>>>>> globalVariableOfCoefficients0 = coefficients0;
>>>>>>>>> globalVariableOfCoefficients1 = null;
>>>>>>>>> 
>>>>>>>>> audioBufferSourceNode.start(0);
>>>>>>>>> 
>>>>>>>>> The reason to have two scriptProcessorNodes is because I want to do a
>>>>>>>>> smooth
>>>>>>>>> transition of the coefficients, so I do a crossfading between the
>>>>>>>>> 'old'
>>>>>>>>> coefficients (scriptProcessorNode0) and the 'new' coefficients
>>>>>>>>> (scriptProcessorNode1) with the ramps of gainNode0 and gainNode1. So
>>>>>>>>> when I
>>>>>>>>> receive the notification to update the coefficients, the global
>>>>>>>>> variable is
>>>>>>>>> updated and the ramps are started.
>>>>>>>>> The first problem is that when I change the
>>>>>>>>> globalVariableOfCoefficients1, I
>>>>>>>>> don't know if the value of the variable is really updated in the
>>>>>>>>> scriptProcessorNode. It seems that the scriptProcessorNode have to
>>>>>>>>> wait
>>>>>>>>> until get a new buffer to update the value of their global variables
>>>>>>>>> . On
>>>>>>>>> the other hand, there a second problem. If I change the value of the
>>>>>>>>> globalVariableOfCoefficients1 and I wait to get a new buffer for
>>>>>>>>> update
>>>>>>>>> their global variables, how I can know when the first sample of this
>>>>>>>>> new
>>>>>>>>> buffer "is" really in the gainNode?
>>>>>>>>> 
>>>>>>>>> On the other hand, I would like to find some documentation where the
>>>>>>>>> relation between the scriptProcessorNode and the audio thread  is
>>>>>>>>> explained
>>>>>>>>> for clearly understand the problematic.
>>>>>>>>> 
>>>>>>>>> Thank you very much in advance,
>>>>>>>>> 
>>>>>>>>> Arnau Julià
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> <diagram_webAudio.png>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Lonce Wyse Dept. of Communications and New Media National University of
>>>>>> Singapore
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>