- From: Chris Wilson <cwilso@google.com>
- Date: Wed, 7 May 2014 11:13:22 -0700
- To: "Srikumar K. S." <srikumarks@gmail.com>
- Cc: Joseph Berkovitz <joe@noteflight.com>, Arnau Julia <Arnau.Julia@ircam.fr>, lonce.audio@sonic.zwhome.org, "public-audio-dev@w3.org" <public-audio-dev@w3.org>
- Message-ID: <CAJK2wqWMTBJKQdhYmHTyqicxf9q9p3m2hM+r3ogUbJXV5dDsag@mail.gmail.com>
Although this is definitely still an issue (Issue #12, as a matter of fact! https://github.com/WebAudio/web-audio-api/issues/12), I would like to caution that we cannot necessarily fix this entirely. IIRC, in a number of cases, we simply do not know what latency is caused by the hardware device itself; I think we can only account for the buffering latency in our own systems. (e.g., your Bluetooth example - I'm not sure there's a way to detect that latency!) On Tue, May 6, 2014 at 9:54 PM, Srikumar K. S. <srikumarks@gmail.com> wrote: > There is also a different "sync" issue that is yet to be addressed. > Currently, we do not have a way to translate between a time expressed in > the AudioContext.currentTime coordinates into the DOMHighResTimeStamp > coordinates. Times in requestAnimationFrame are DOMHighResTimeStamp times > (iirc) and synchronizing visuals with computed audio is near impossible > without a straightforward way to translate between them. This gets worse on > mobile devices where a bluetooth speaker can get connected while an audio > context is running and on-the-fly add 300ms of latency. > > I don't think I've missed any development towards this, but if I have, I > apologize for raising this again and am all ears to hear the solution. > > -Kumar > sriku.org > > > On 7 May, 2014, at 12:36 am, Joseph Berkovitz <joe@noteflight.com> wrote: > > To echo Chris W, it is *possible* to sync by paying attention to the > playbackTime of an audio processing event, and by scheduling parameter > changes and actions on native nodes in relation to that time (which is > expressed in the AudioContext timebase). > > However, due to the fact that the code in a ScriptProcessorNode runs in > the main JS thread, as a practical matter it is difficult to do such > syncing reliably and robustly without glitching. There are also some > browser portability issues as Chris mentioned. > > Hence the urgency to find a better solution. > > . . . . . ...Joe > > *Joe Berkovitz* > President > > *Noteflight LLC* > Boston, Mass. > phone: +1 978 314 6271 > www.noteflight.com > "Your music, everywhere" > > On May 6, 2014, at 4:13 AM, Arnau Julia <Arnau.Julia@ircam.fr> wrote: > > Hello, > > I'm aware of public-audio list conversations about the use of workers for > the scriptProcessorNode and I'm very excited about the possibilities of > this solution, but I supposed that it was possible to sync a > scriptProcessorNode and a native Node with the current implementation. Am I > wrong? And if not, how it is possible to achieve? > > Thank you, > > Arnau > > On 5 mai 2014, at 18:42, Chris Wilson <cwilso@google.com> wrote: > > Lonce, > > this is one of the biggest and most important issues on my Web Audio plate > right now. I'm working on figuring out how to spark some coming together > of implementers over the summer to come up with a workable solution. > > > On Fri, May 2, 2014 at 9:38 PM, lonce <lonce.audio@sonic.zwhome.org>wrote: > >> >> Hi - >> >> I think the real question is not how to hack this, but the status of >> progress on a fundamental solution to this Achilles heal of the current >> system. From what I gather, the solution will probably be in the form of >> web workers (?), but I don't know how much attention this is getting now. >> Once this is solved, the system becomes truly extensible and I am >> sure it will open up an explosive era of community development just waiting >> to happen! >> >> Best, >> - lonce >> >> >> On 5/2/2014 4:39 PM, Arnau Julia wrote: >> >>> Hello, >>> >>> First of all, thank for all your answers. >>> >>> The first thing to note is that all script processor node processing >>>> happens on the main javascript thread. This means if you change a >>>> global variable in another part of your javascript program, it will >>>> show definitely show up on the next AudioProcessingEvent. So, that >>>> answers your first problem - once you set the variable in your >>>> javascript, on the next buffer the change will be there. There's no >>>> parallelism at all in the javascript - there's only one thing happening >>>> at once. >>>> >>> I would like to understand how it works. The difference that I found >>> between the scriptProcessorNode and the 'native' AudioNode Interface is >>> that the first uses a Event Handler and the AudioNodes are EventTargets. Is >>> it the reason why the global variables are updated only one time for each >>> buffer? Someone have more documentation to understand it more deeply? >>> >>> For your second question, you need some sort of timestamp on the >>>> buffer. The web audio api provides this as the playbackTime field on >>>> the AudioProcessingEvent. Of course, you only have access to the >>>> playback time of the buffer you are currently processing, but you can >>>> guess when the next playbackTime will be by setting the last processed >>>> time as a global variable, and then adding one buffer's worth of time >>>> to that to get the next playbackTime. This will be fine unless you >>>> drop buffers, in which case you're probably not worried about a smooth >>>> ramp :-). So, one easy solution to your second problem is to always >>>> store the last playback time that each of your script nodes processed, >>>> and then start the ramp on the *next* buffer. The spec guarantees that >>>> the playbackTime and ramping is sample accurate, so no worries >>>> there. In practice, the last time I checked, which was over a year >>>> ago, firefox had serious problems with the playbackTime field (I don't >>>> remember if it was just absent or if it had some other problem that >>>> made it unusable.) >>>> >>> It seems a good solution! I didn't found the playbackTime on the last >>> stable version of Chrome but I found it in Firefox. Is there any >>> alternative for Chrome? >>> >>> I have done some basic experiments with playbackTime in Firefox and it >>> seems that is not totally sync or maybe I don't understand how to use it. I >>> uploaded the experiment to jsfiddle (only Firefox!): >>> http://jsfiddle.net/PgeLv/11/ >>> The experiment structure is: >>> oscillatorNode (source) ----> scriptProcesorNode -----> GainNode >>> -------> Destination >>> >>> On the other hand, I would like to understand 'what' is exactly the >>> playbackTime. I guess that it can be something like that: >>> >>> playbackTime = bufferSize/sampleRate + 'processTime' + 'wait interval >>> until the event return the data to the audio thread' >>> >>> If this hypothesis is true, it means that the playbackTime is different >>> for each event, because it depends on the activity of the general thread. >>> >>> Thanks, >>> >>> Arnau >>> >>> On 22 avr. 2014, at 01:51, Russell McClellan < >>> russell.mcclellan@gmail.com> wrote: >>> >>> Hey Arnau - >>>> >>>> Yes, this is probably underdocumented. The good news is, the >>>> designers of the web audio api do actually have an answer for linking >>>> native nodes and script processor nodes. >>>> >>>> The first thing to note is that all script processor node processing >>>> happens on the main javascript thread. This means if you change a >>>> global variable in another part of your javascript program, it will >>>> show definitely show up on the next AudioProcessingEvent. So, that >>>> answers your first problem - once you set the variable in your >>>> javascript, on the next buffer the change will be there. There's no >>>> parallelism at all in the javascript - there's only one thing >>>> happening at once. >>>> >>>> For your second question, you need some sort of timestamp on the >>>> buffer. The web audio api provides this as the playbackTime field on >>>> the AudioProcessingEvent. Of course, you only have access to the >>>> playback time of the buffer you are currently processing, but you can >>>> guess when the next playbackTime will be by setting the last processed >>>> time as a global variable, and then adding one buffer's worth of time >>>> to that to get the next playbackTime. This will be fine unless you >>>> drop buffers, in which case you're probably not worried about a smooth >>>> ramp :-). So, one easy solution to your second problem is to always >>>> store the last playback time that each of your script nodes processed, >>>> and then start the ramp on the *next* buffer. The spec guarantees >>>> that the playbackTime and ramping is sample accurate, so no worries >>>> there. In practice, the last time I checked, which was over a year >>>> ago, firefox had serious problems with the playbackTime field (I don't >>>> remember if it was just absent or if it had some other problem that >>>> made it unusable.) >>>> >>>> Thanks, >>>> -Russell >>>> >>>> On Fri, Apr 18, 2014 at 10:50 AM, Casper Schipper >>>> <casper.schipper@monotonestudio.nl> wrote: >>>> >>>>> Dear Arnau, >>>>> >>>>> this is indeed a frustrating (but probably performance wise necessary) >>>>> limitation of the normal web audio nodes, >>>>> parameters in a scriptProcessorNode can only be updated once every >>>>> vector >>>>> which is a minimum of 256 samples. >>>>> >>>>> Maybe you could solve your problem by using one of the javascript >>>>> libraries >>>>> that bypass most of web audio api and do everything in JS itself. >>>>> What comes first to mind would be the Gibberish.js library by Charlie >>>>> Roberts, which gives you the ability to control parameters per sample >>>>> and >>>>> easily schedule synchronized parameter changes also with sample >>>>> accuracy: >>>>> http://www.charlie-roberts.com/gibberish/docs.html >>>>> It should be quite easy to extend it with your own nodes. >>>>> There are other libraries as well like flocking.js and Timbre.js. >>>>> >>>>> Of course it comes with some performance penalties, but Gibberish >>>>> tries to >>>>> at least generate javascript code that should be as efficient as >>>>> possible >>>>> for it's JIT complication style, as far as it's own nodes are >>>>> considered. >>>>> >>>>> Hope it helps, >>>>> Casper >>>>> >>>>> casper.schipper@monotonestudio.nl >>>>> Mauritskade 55C (the thinking hut) >>>>> 1092 AD Amsterdam >>>>> +316 52 322 590 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 18 apr. 2014, at 10:55, Arnau Julia <Arnau.Julia@ircam.fr> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I'm trying to synchronizing the buffer in a scriptProcessorNode with >>>>> native/regular web audio nodes and I'm having some problems. My >>>>> problem is >>>>> that I want to synchronize the scriptProcessorNode with a ramp of a >>>>> GainNode. >>>>> >>>>> My program looks like the attached diagram. Each scriptProcessorNode >>>>> is a >>>>> filter with n coefficients and these coefficients are in a global >>>>> variable. >>>>> My problem comes when I try to update these coefficients and do a ramp >>>>> in >>>>> the gain through an audioParam at the "same time". >>>>> >>>>> The start scenario is (in pseudo-code): >>>>> >>>>> audioBufferSourceNode.connect(scriptProcessorNode0); >>>>> audioBufferSourceNode.connect(scriptProcessorNode1); >>>>> >>>>> scriptProcessorNode0.connect(gainNode0); >>>>> scriptProcessorNode0.connect(gainNode1); >>>>> >>>>> gainNode0.connect(audioContext.destination); >>>>> gainNode1.connect(audioContext.destination); >>>>> >>>>> gainNode1.gain.value = 0; >>>>> globalVariableOfCoefficients0 = coefficients0; >>>>> globalVariableOfCoefficients1 = null; >>>>> >>>>> audioBufferSourceNode.start(0); >>>>> >>>>> The reason to have two scriptProcessorNodes is because I want to do a >>>>> smooth >>>>> transition of the coefficients, so I do a crossfading between the 'old' >>>>> coefficients (scriptProcessorNode0) and the 'new' coefficients >>>>> (scriptProcessorNode1) with the ramps of gainNode0 and gainNode1. So >>>>> when I >>>>> receive the notification to update the coefficients, the global >>>>> variable is >>>>> updated and the ramps are started. >>>>> The first problem is that when I change the >>>>> globalVariableOfCoefficients1, I >>>>> don't know if the value of the variable is really updated in the >>>>> scriptProcessorNode. It seems that the scriptProcessorNode have to wait >>>>> until get a new buffer to update the value of their global variables . >>>>> On >>>>> the other hand, there a second problem. If I change the value of the >>>>> globalVariableOfCoefficients1 and I wait to get a new buffer for update >>>>> their global variables, how I can know when the first sample of this >>>>> new >>>>> buffer "is" really in the gainNode? >>>>> >>>>> On the other hand, I would like to find some documentation where the >>>>> relation between the scriptProcessorNode and the audio thread is >>>>> explained >>>>> for clearly understand the problematic. >>>>> >>>>> Thank you very much in advance, >>>>> >>>>> Arnau JuliĆ >>>>> >>>>> >>>>> <diagram_webAudio.png> >>>>> >>>>> >>>>> >>>> >>>> >>> >> >> -- >> Lonce Wyse Dept. of Communications and New Media National University of >> Singapore >> >> > > > >
Received on Wednesday, 7 May 2014 18:13:51 UTC