W3C home > Mailing lists > Public > public-audio@w3.org > October to December 2012

Re: MIDI synchronization issues

From: James Ingram <j.ingram@netcologne.de>
Date: Thu, 04 Oct 2012 00:19:27 +0200
Message-ID: <506CB9EF.3070802@netcologne.de>
To: Chris Wilson <cwilso@google.com>
CC: Srikumar Karaikudi Subramanian <srikumarks@gmail.com>, public-audio@w3.org, Jussi Kalliokoski <jussi.kalliokoski@gmail.com>, Joseph Berkovitz <joe@noteflight.com>

Hi Chris, Kumar, all

Thanks Chris and Kumar for looking at this. Sorry about the belated reply.

Chris said:
> There will not be any way to get a precise timestamp of when a MIDI 
> message is ACTUALLY sent; the underlying hardware may not report that, 
> and we don't have any callback mechanism that has a 
> super-high-precision timing itself (below 1ms, and the slop in 
> setTimeout/Interval is, as you've found, >1ms).
But I absolutely MUST report certain timestamps back to the GUI, 
otherwise its going to hang when it loses track of where it is. I can't 
just let the timestamps go unreported.
The answer I've found is to mark the messages I need to track (by giving 
them a "reportTimestamp" attribute, and then cache their timestamps 
during the PREQUEUE chunk. Timestamps thus cached are simply reported to 
the GUI when the chunk ends. To be clear, that means that the timestamps 
are reported back to the GUI before their messages are actually sent to 
the output device. The maximum time difference between advancing the 
cursor and hearing the sound should be PREQUEUE milliseconds.

This still needs testing with a conformant sendMIDIMessage (which 
respects timestamps) and a larger value for PREQUEUE. But even with 
PREQUEUE at zero, this approach is more robust than simply ignoring all 
timestamps inside the PREQUEUE loop. It has certainly made my code a bit 
more future-proof. :-)

This exercise has certainly convinced me that while all Sequence objects 
might have very similar interfaces, their implementation is something 
that ought to be left to individual programmers/libraries.

Kumar suggested using strictly periodic calls to tick. That may well be 
a simpler and correct approach in many situations, but in my case 
(Chris' original idea) the 'delay' variable compensates very nicely for 
the inaccuracies in setTimeout. Also, I can't see how I would capture my 
'msg' variables often enough. The differences between the timeouts in 
the msgs can be much smaller than 60Hz, and I'm not seeing any great 
discrepancies between cursor movements and sounds using the current 
approach, so I think I'll stick with it.

All the best,
James

p.s. My project has now gone open-source at
https://github.com/notator/assistant-performer
The tick function is in js/jiSequence.js




> Srikumar,
>
> yes, that's what I had in mind with this approach initially.  You need 
> a callback that's going to occur some amount of time BEFORE the next 
> prequeue "chunk" is due.
>
> On your [1] - yes, they are unreliable; that's the major reason we 
> even have timestamps in sending MIDI messages.
>
> On Mon, Oct 1, 2012 at 5:20 PM, Srikumar Karaikudi Subramanian 
> <srikumarks@gmail.com <mailto:srikumarks@gmail.com>> wrote:
>
>     Hi James,
>
>     The tick() function in the gist is not the right approach, I
>     think. If we consider a MIDI sequence that has drum hits spaced 1
>     sec apart and use PREQUEUE = 500ms (anything less than 1 sec
>     actually) , then the tick() will try to land itself just in time
>     to send out the drum hit, by which time it may already be too late
>     [1] ... unless I'm reading the gist code wrongly.
>
>     A better approach would be to ask the system for *regular* tick()
>     callbacks with a period that is *independent* of the message time
>     stamps. Then if we have a PREQUEUE that is some multiple (ex: 3x)
>     of the tick() interval, we can catch up if a large jitter happens
>     and the time stamps become useful to the backend high precision
>     scheduler.
>
>     The best way to get reliable periodic callbacks in a browser is to
>     use requestAnimationFrame(). If my reading of your gist code is
>     right, all that is needed is to change the last line from -
>             window.setTimeout(tick, delay);
>     to -
>             webkitRequestAnimationFrame(tick); // Gets called at 60Hz
>     usually.
>
>     Then, when time stamps begin to be supported in the MIDI
>     implementation, PREQUE = 50ms can give the rock solid timing
>     promised by the underlying implementation. This will also allow a
>     JS player to respond to global tempo changes quickly if there is a
>     tempo slider in the UI.
>
>     Furthermore, in current browsers, it is not possible to guarantee
>     a small enough upper limit on the time difference between a
>     generated event and a JS function called in response. This means
>     that even if you have a "MIDI message sent" callback installed,
>     you may receive that late anyway. It is easier (imho) to assume
>     that the message goes out as expected and plan to do the necessary
>     UI. If the UI is wrong, it would be wrong only up to PREQUEUE.
>
>     Best,
>     -Kumar
>
>     [1] setInterval and setTimeout are just plain unreliable for music
>     timing in the browser as far as I've seen. I've measured standard
>     deviations of 400ms for such "target times" in Chrome some time
>     ago. In Node.js, though, setInterval is rock solid (deviations ~<
>     0.4ms) while setTimeout is as goofy as in Chrome. (Node.js has a
>     MIDI npm module btw.)
>
>     On 2 Oct, 2012, at 2:29 AM, James Ingram <j.ingram@netcologne.de
>     <mailto:j.ingram@netcologne.de>> wrote:
>
>     > Hi Chris,
>     >
>     > I'll put it another way: There is going to be a synchronization
>     problem when the timestamp property on sendMiDIMessage is
>     implemented and PREQUEUE is increased.
>     >
>     > Lets say the sendMIDIMessage timestamps are working as
>     advertised, and we increase PREQUEUE to 500. That would mean in
>     [1] that the reportTimestamp function would only be called ca.
>     every 500ms, and would not be able to report the timestamp as
>     often as it needs to. There is no way, as things stand, for the
>     world outside the while{} loop to report the sending of events
>     inside the loop at the time they are actually sent. I think this
>     problem needs thinking about now. We don't need to wait for
>     sendMIDIMessage timestamps to be implemented.
>     >
>     > Whatever the answer is going to be, I think it also needs to
>     take empty messages into account. Empty messages have lots of uses
>     when synchronizing with other things going on on the same
>     computer. Maybe one of MIDI's more obscure messages could be
>     hijacked for the purpose, but the state of the output device
>     should not change. Maybe someone has an idea in that direction?
>     >
>     > Hope that's a bit clearer.
>     >
>     > best,
>     > James
>     >
>     > [1] https://gist.github.com/3806262
>
>


-- 
www.james-ingram-act-two.de
Received on Wednesday, 3 October 2012 22:20:17 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:50:03 UTC