- From: Srikumar K. S. <srikumarks@gmail.com>
- Date: Sat, 17 May 2014 07:11:30 +0530
- To: Russell McClellan <russell.mcclellan@gmail.com>
- Cc: Chris Wilson <cwilso@google.com>, Joseph Berkovitz <joe@noteflight.com>, Arnau Julia <Arnau.Julia@ircam.fr>, lonce.audio@sonic.zwhome.org, "public-audio-dev@w3.org" <public-audio-dev@w3.org>
- Message-Id: <4B6B43A4-5E87-4E9C-92D5-2DAD14BFFBBF@gmail.com>
> Chris - Yes, windows (at least the newer WASAPI) reports the stream > latency (a function called "GetStreamLatency" on the audio client). The thread [1] mentions IAudioClock::GetPosition as possibly a more accurate way to get to the latency info. However, the description in the MSDN man page [2] is not as direct as the description in the thread. [1] http://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/c4fd959b-a0c8-40a0-b037-f949c2428337/audio-inputoutput-synchronization-under-wasapi "IAudioClock::GetPosition will tell you the sample that is hitting the air at the speaker cone (for render) or is coming in from the air (for the microphone.)" [2] http://msdn.microsoft.com/en-us/library/windows/desktop/dd370889(v=vs.85).aspx -Kumar On 17 May, 2014, at 6:37 am, Russell McClellan <russell.mcclellan@gmail.com> wrote: > Chris - Yes, windows (at least the newer WASAPI) reports the stream > latency (a function called "GetStreamLatency" on the audio client). > Linux is trickier and I'm not sure about it. All professional DAWs > for mac and windows display the output latency of the interface. > > Also, I've said this before, but as a web developer, script processors > in workers would be a total godsend. It would make my life a *lot* > easier. > > Arnau - > > First of all, I think your test should work, and the bug is in the > browser (although, it would be better if you had set the gain to 4 in > the second stage, so that a correct browser would play an unbroken > sine wave). As I mentioned previously, I have never had any success > with playbackTime in firefox. In chrome, it at least used to work, I > believe. > > There's very little high-level documentation here, so I think you may > be a bit confused about the general architecture of the web audio api. > When you get an "audio processing event", you are in the main > javascript thread. The audio processing callback is called using > literally the same mechanism as any other event - clicks, or whatever. > There's nothing concurrent or different about audio processing > callbacks than any other sort of event handler. That's why the global > variable update works. It's just like if a user was clicking on a > button 100 times a second or so. You could update a global variable > on a timer, but while you're in the click handler, you wouldn't see > the global variable update until you returned and were called again. > > So, the audio system has determined that it needs a buffer from you > that will be played at a specific time, and it is your job to fill up > that buffer's audio. When you return from the event callback, that > audio will be transferred to the audio system. The audio system has > already determined when the buffer you are working on should be > played, and that's the playbackTime timestamp. If you return from the > event callback in time for the audio system to play back that buffer > before playbackTime, the buffer will be played at playbackTime. If > you don't return from the callback in time, who knows what will > happen? (I'm not sure if this is even specified). > > As far as how playbackTime is calculated, that's not really your > concern, as you're not the one implementing the audio system. It > should be fairly complicated. If everything is going smoothly though, > it should at least increment by one buffer every callback - otherwise, > something is dropping audio somewhere. It has no bearing at all on > the time your audio callback is called (is this what you mean by > "processTime"?)- it only refers to the time that the buffer will be > played. > > Thanks, > -Russell > > > > > On Thu, May 8, 2014 at 4:09 PM, Chris Wilson <cwilso@google.com> wrote: >> Hmm, interesting. Any Windows/Linux sound api consumers want to contribute >> here? (I couldn't find a comparable property in Windows APIs with a quick >> look.) >> >> >> On Wed, May 7, 2014 at 9:58 PM, Srikumar K. S. <srikumarks@gmail.com> wrote: >>> >>> (e.g., your Bluetooth example - I'm not sure there's a way to detect that >>> latency!) >>> >>> >>> There is ... at least in iOS and MacOSX. I use it in my iOS app. When the >>> audio route changes, I just ask for the "CurrentHardwareOutputLatency" >>> property >>> of the AudioSession. >>> >>> Even if the internal buffering is the only latency that the system can >>> access, >>> that would still be useful to have explicitly via the API than not have >>> it. >>> This would permit the API implementations to account for such latency >>> info where and when it is available. >>> >>> -Kumar >>> >>> On 7 May, 2014, at 11:43 pm, Chris Wilson <cwilso@google.com> wrote: >>> >>> Although this is definitely still an issue (Issue #12, as a matter of >>> fact! https://github.com/WebAudio/web-audio-api/issues/12), I would like to >>> caution that we cannot necessarily fix this entirely. IIRC, in a number of >>> cases, we simply do not know what latency is caused by the hardware device >>> itself; I think we can only account for the buffering latency in our own >>> systems. (e.g., your Bluetooth example - I'm not sure there's a way to >>> detect that latency!) >>> >>> >>> On Tue, May 6, 2014 at 9:54 PM, Srikumar K. S. <srikumarks@gmail.com> >>> wrote: >>>> >>>> There is also a different "sync" issue that is yet to be addressed. >>>> Currently, we do not have a way to translate between a time expressed in the >>>> AudioContext.currentTime coordinates into the DOMHighResTimeStamp >>>> coordinates. Times in requestAnimationFrame are DOMHighResTimeStamp times >>>> (iirc) and synchronizing visuals with computed audio is near impossible >>>> without a straightforward way to translate between them. This gets worse on >>>> mobile devices where a bluetooth speaker can get connected while an audio >>>> context is running and on-the-fly add 300ms of latency. >>>> >>>> I don't think I've missed any development towards this, but if I have, I >>>> apologize for raising this again and am all ears to hear the solution. >>>> >>>> -Kumar >>>> sriku.org >>>> >>>> >>>> On 7 May, 2014, at 12:36 am, Joseph Berkovitz <joe@noteflight.com> wrote: >>>> >>>> To echo Chris W, it is *possible* to sync by paying attention to the >>>> playbackTime of an audio processing event, and by scheduling parameter >>>> changes and actions on native nodes in relation to that time (which is >>>> expressed in the AudioContext timebase). >>>> >>>> However, due to the fact that the code in a ScriptProcessorNode runs in >>>> the main JS thread, as a practical matter it is difficult to do such syncing >>>> reliably and robustly without glitching. There are also some browser >>>> portability issues as Chris mentioned. >>>> >>>> Hence the urgency to find a better solution. >>>> >>>> . . . . . ...Joe >>>> >>>> Joe Berkovitz >>>> President >>>> >>>> Noteflight LLC >>>> Boston, Mass. phone: +1 978 314 6271 >>>> www.noteflight.com >>>> "Your music, everywhere" >>>> >>>> On May 6, 2014, at 4:13 AM, Arnau Julia <Arnau.Julia@ircam.fr> wrote: >>>> >>>> Hello, >>>> >>>> I'm aware of public-audio list conversations about the use of workers for >>>> the scriptProcessorNode and I'm very excited about the possibilities of this >>>> solution, but I supposed that it was possible to sync a scriptProcessorNode >>>> and a native Node with the current implementation. Am I wrong? And if not, >>>> how it is possible to achieve? >>>> >>>> Thank you, >>>> >>>> Arnau >>>> >>>> On 5 mai 2014, at 18:42, Chris Wilson <cwilso@google.com> wrote: >>>> >>>> Lonce, >>>> >>>> this is one of the biggest and most important issues on my Web Audio >>>> plate right now. I'm working on figuring out how to spark some coming >>>> together of implementers over the summer to come up with a workable >>>> solution. >>>> >>>> >>>> On Fri, May 2, 2014 at 9:38 PM, lonce <lonce.audio@sonic.zwhome.org> >>>> wrote: >>>>> >>>>> >>>>> Hi - >>>>> >>>>> I think the real question is not how to hack this, but the status of >>>>> progress on a fundamental solution to this Achilles heal of the current >>>>> system. From what I gather, the solution will probably be in the form of web >>>>> workers (?), but I don't know how much attention this is getting now. >>>>> Once this is solved, the system becomes truly extensible and I am >>>>> sure it will open up an explosive era of community development just waiting >>>>> to happen! >>>>> >>>>> Best, >>>>> - lonce >>>>> >>>>> >>>>> On 5/2/2014 4:39 PM, Arnau Julia wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> First of all, thank for all your answers. >>>>>> >>>>>>> The first thing to note is that all script processor node processing >>>>>>> happens on the main javascript thread. This means if you change a >>>>>>> global variable in another part of your javascript program, it will >>>>>>> show definitely show up on the next AudioProcessingEvent. So, that >>>>>>> answers your first problem - once you set the variable in your >>>>>>> javascript, on the next buffer the change will be there. There's no >>>>>>> parallelism at all in the javascript - there's only one thing >>>>>>> happening at once. >>>>>> >>>>>> I would like to understand how it works. The difference that I found >>>>>> between the scriptProcessorNode and the 'native' AudioNode Interface is that >>>>>> the first uses a Event Handler and the AudioNodes are EventTargets. Is it >>>>>> the reason why the global variables are updated only one time for each >>>>>> buffer? Someone have more documentation to understand it more deeply? >>>>>> >>>>>>> For your second question, you need some sort of timestamp on the >>>>>>> buffer. The web audio api provides this as the playbackTime field on >>>>>>> the AudioProcessingEvent. Of course, you only have access to the >>>>>>> playback time of the buffer you are currently processing, but you can >>>>>>> guess when the next playbackTime will be by setting the last processed >>>>>>> time as a global variable, and then adding one buffer's worth of time >>>>>>> to that to get the next playbackTime. This will be fine unless you >>>>>>> drop buffers, in which case you're probably not worried about a smooth >>>>>>> ramp :-). So, one easy solution to your second problem is to always >>>>>>> store the last playback time that each of your script nodes processed, >>>>>>> and then start the ramp on the *next* buffer. The spec guarantees >>>>>>> that the playbackTime and ramping is sample accurate, so no worries >>>>>>> there. In practice, the last time I checked, which was over a year >>>>>>> ago, firefox had serious problems with the playbackTime field (I don't >>>>>>> remember if it was just absent or if it had some other problem that >>>>>>> made it unusable.) >>>>>> >>>>>> It seems a good solution! I didn't found the playbackTime on the last >>>>>> stable version of Chrome but I found it in Firefox. Is there any alternative >>>>>> for Chrome? >>>>>> >>>>>> I have done some basic experiments with playbackTime in Firefox and it >>>>>> seems that is not totally sync or maybe I don't understand how to use it. I >>>>>> uploaded the experiment to jsfiddle (only Firefox!): >>>>>> http://jsfiddle.net/PgeLv/11/ >>>>>> The experiment structure is: >>>>>> oscillatorNode (source) ----> scriptProcesorNode -----> GainNode >>>>>> -------> Destination >>>>>> >>>>>> On the other hand, I would like to understand 'what' is exactly the >>>>>> playbackTime. I guess that it can be something like that: >>>>>> >>>>>> playbackTime = bufferSize/sampleRate + 'processTime' + 'wait interval >>>>>> until the event return the data to the audio thread' >>>>>> >>>>>> If this hypothesis is true, it means that the playbackTime is different >>>>>> for each event, because it depends on the activity of the general thread. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Arnau >>>>>> >>>>>> On 22 avr. 2014, at 01:51, Russell McClellan >>>>>> <russell.mcclellan@gmail.com> wrote: >>>>>> >>>>>>> Hey Arnau - >>>>>>> >>>>>>> Yes, this is probably underdocumented. The good news is, the >>>>>>> designers of the web audio api do actually have an answer for linking >>>>>>> native nodes and script processor nodes. >>>>>>> >>>>>>> The first thing to note is that all script processor node processing >>>>>>> happens on the main javascript thread. This means if you change a >>>>>>> global variable in another part of your javascript program, it will >>>>>>> show definitely show up on the next AudioProcessingEvent. So, that >>>>>>> answers your first problem - once you set the variable in your >>>>>>> javascript, on the next buffer the change will be there. There's no >>>>>>> parallelism at all in the javascript - there's only one thing >>>>>>> happening at once. >>>>>>> >>>>>>> For your second question, you need some sort of timestamp on the >>>>>>> buffer. The web audio api provides this as the playbackTime field on >>>>>>> the AudioProcessingEvent. Of course, you only have access to the >>>>>>> playback time of the buffer you are currently processing, but you can >>>>>>> guess when the next playbackTime will be by setting the last processed >>>>>>> time as a global variable, and then adding one buffer's worth of time >>>>>>> to that to get the next playbackTime. This will be fine unless you >>>>>>> drop buffers, in which case you're probably not worried about a smooth >>>>>>> ramp :-). So, one easy solution to your second problem is to always >>>>>>> store the last playback time that each of your script nodes processed, >>>>>>> and then start the ramp on the *next* buffer. The spec guarantees >>>>>>> that the playbackTime and ramping is sample accurate, so no worries >>>>>>> there. In practice, the last time I checked, which was over a year >>>>>>> ago, firefox had serious problems with the playbackTime field (I don't >>>>>>> remember if it was just absent or if it had some other problem that >>>>>>> made it unusable.) >>>>>>> >>>>>>> Thanks, >>>>>>> -Russell >>>>>>> >>>>>>> On Fri, Apr 18, 2014 at 10:50 AM, Casper Schipper >>>>>>> <casper.schipper@monotonestudio.nl> wrote: >>>>>>>> >>>>>>>> Dear Arnau, >>>>>>>> >>>>>>>> this is indeed a frustrating (but probably performance wise >>>>>>>> necessary) >>>>>>>> limitation of the normal web audio nodes, >>>>>>>> parameters in a scriptProcessorNode can only be updated once every >>>>>>>> vector >>>>>>>> which is a minimum of 256 samples. >>>>>>>> >>>>>>>> Maybe you could solve your problem by using one of the javascript >>>>>>>> libraries >>>>>>>> that bypass most of web audio api and do everything in JS itself. >>>>>>>> What comes first to mind would be the Gibberish.js library by Charlie >>>>>>>> Roberts, which gives you the ability to control parameters per sample >>>>>>>> and >>>>>>>> easily schedule synchronized parameter changes also with sample >>>>>>>> accuracy: >>>>>>>> http://www.charlie-roberts.com/gibberish/docs.html >>>>>>>> It should be quite easy to extend it with your own nodes. >>>>>>>> There are other libraries as well like flocking.js and Timbre.js. >>>>>>>> >>>>>>>> Of course it comes with some performance penalties, but Gibberish >>>>>>>> tries to >>>>>>>> at least generate javascript code that should be as efficient as >>>>>>>> possible >>>>>>>> for it's JIT complication style, as far as it's own nodes are >>>>>>>> considered. >>>>>>>> >>>>>>>> Hope it helps, >>>>>>>> Casper >>>>>>>> >>>>>>>> casper.schipper@monotonestudio.nl >>>>>>>> Mauritskade 55C (the thinking hut) >>>>>>>> 1092 AD Amsterdam >>>>>>>> +316 52 322 590 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 18 apr. 2014, at 10:55, Arnau Julia <Arnau.Julia@ircam.fr> wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I'm trying to synchronizing the buffer in a scriptProcessorNode with >>>>>>>> native/regular web audio nodes and I'm having some problems. My >>>>>>>> problem is >>>>>>>> that I want to synchronize the scriptProcessorNode with a ramp of a >>>>>>>> GainNode. >>>>>>>> >>>>>>>> My program looks like the attached diagram. Each scriptProcessorNode >>>>>>>> is a >>>>>>>> filter with n coefficients and these coefficients are in a global >>>>>>>> variable. >>>>>>>> My problem comes when I try to update these coefficients and do a >>>>>>>> ramp in >>>>>>>> the gain through an audioParam at the "same time". >>>>>>>> >>>>>>>> The start scenario is (in pseudo-code): >>>>>>>> >>>>>>>> audioBufferSourceNode.connect(scriptProcessorNode0); >>>>>>>> audioBufferSourceNode.connect(scriptProcessorNode1); >>>>>>>> >>>>>>>> scriptProcessorNode0.connect(gainNode0); >>>>>>>> scriptProcessorNode0.connect(gainNode1); >>>>>>>> >>>>>>>> gainNode0.connect(audioContext.destination); >>>>>>>> gainNode1.connect(audioContext.destination); >>>>>>>> >>>>>>>> gainNode1.gain.value = 0; >>>>>>>> globalVariableOfCoefficients0 = coefficients0; >>>>>>>> globalVariableOfCoefficients1 = null; >>>>>>>> >>>>>>>> audioBufferSourceNode.start(0); >>>>>>>> >>>>>>>> The reason to have two scriptProcessorNodes is because I want to do a >>>>>>>> smooth >>>>>>>> transition of the coefficients, so I do a crossfading between the >>>>>>>> 'old' >>>>>>>> coefficients (scriptProcessorNode0) and the 'new' coefficients >>>>>>>> (scriptProcessorNode1) with the ramps of gainNode0 and gainNode1. So >>>>>>>> when I >>>>>>>> receive the notification to update the coefficients, the global >>>>>>>> variable is >>>>>>>> updated and the ramps are started. >>>>>>>> The first problem is that when I change the >>>>>>>> globalVariableOfCoefficients1, I >>>>>>>> don't know if the value of the variable is really updated in the >>>>>>>> scriptProcessorNode. It seems that the scriptProcessorNode have to >>>>>>>> wait >>>>>>>> until get a new buffer to update the value of their global variables >>>>>>>> . On >>>>>>>> the other hand, there a second problem. If I change the value of the >>>>>>>> globalVariableOfCoefficients1 and I wait to get a new buffer for >>>>>>>> update >>>>>>>> their global variables, how I can know when the first sample of this >>>>>>>> new >>>>>>>> buffer "is" really in the gainNode? >>>>>>>> >>>>>>>> On the other hand, I would like to find some documentation where the >>>>>>>> relation between the scriptProcessorNode and the audio thread is >>>>>>>> explained >>>>>>>> for clearly understand the problematic. >>>>>>>> >>>>>>>> Thank you very much in advance, >>>>>>>> >>>>>>>> Arnau Julià >>>>>>>> >>>>>>>> >>>>>>>> <diagram_webAudio.png> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Lonce Wyse Dept. of Communications and New Media National University of >>>>> Singapore >>>>> >>>> >>>> >>>> >>>> >>> >>> >>
Received on Saturday, 17 May 2014 01:42:06 UTC