W3C home > Mailing lists > Public > public-audio@w3.org > July to September 2012

Re: MIDI Tracks and Sequences

From: James Ingram <j.ingram@netcologne.de>
Date: Wed, 12 Sep 2012 23:58:16 +0200
Message-ID: <50510578.5040809@netcologne.de>
To: Chris Wilson <cwilso@google.com>
CC: Jussi Kalliokoski <jussi.kalliokoski@gmail.com>, public-audio@w3.org
Hi Chris,

Sorry, its me that's confused! :-)
> My suggested implementation tactic would be to use setInterval to 
> pre-schedule MIDI events in the send message queue, keeping some 
> arbitrary interval of time (a tenth of a second or so?) ahead of the 
> current DOMHRT time; this slower loop can then be cancelled whenever 
> you hit stop.
I'll try that. Thanks very much for the suggestion. By the way, DOMHRT 
time is AudioContext.currentTime, right?


Am 12.09.2012 19:47, schrieb Chris Wilson:
> James,
> I'm quite confused.  sendTimestampedMIDIMessage in that shim is not a 
> function you should have to write or edit; the Web MIDI API has a 
> "send this message, at this absolute time" capability.  You shouldn't 
> need to deal with clock drift; in a real, native implementation of the 
> Web MIDI API, a lower-level timing system would be handling this, not 
> Javascript's setTimeout/setInterval; much like the Web Audio API has 
> its own system for scheduling.  This is going to be a weak point in 
> any polyfill (the one you point to, and the one I'm writing) - you 
> simply cannot get access to a high-enough resolution clock to properly 
> implement the timestamps, and none of the current ways to access MIDI 
> in Javascript that I know of expose a higher-res timing system.
> For playing/pausing/stopping, the only thing I think we may be missing 
> is the ability to cancel all scheduled MIDI messages, although before 
> imposing that I need to look more closely at how the underlying 
> systems support such a beast; you may need to close the device in 
> order to clear it.
> My suggested implementation tactic would be to use setInterval to 
> pre-schedule MIDI events in the send message queue, keeping some 
> arbitrary interval of time (a tenth of a second or so?) ahead of the 
> current DOMHRT time; this slower loop can then be cancelled whenever 
> you hit stop.
> On Wed, Sep 12, 2012 at 10:10 AM, James Ingram <j.ingram@netcologne.de 
> <mailto:j.ingram@netcologne.de>> wrote:
>     Hi Chris, Jussi, all,
>     As you pointed out, this kind of thing may well belong in a
>     standard library, but I think it might be worth taking a closer
>     look at the problems that are going to have to be solved, in case
>     we discover that lower level solutions might be better after all.
>     As I said, I'm currently writing a sequencer in Javascript, using
>     JazzMIDIBridge [5] to emulate the future Web MIDI API.
>     I've created my own definition of a Sequence, and can now load
>     such Sequences from files. The immediate problem is now to write a
>     simple
>         sequence.play()
>     function which will play the sequence, from beginning to end, as
>     accurately as possible.
>     In general, I expect sequences to be large (there should be no
>     limit, in principle, to their duration), and to contain lots of
>     continuous control data. So I'm creeping up on the problem, by
>     writing a few test files. (I'm working on Windows, and using the
>     latest Chrome browser.)
>     Test 1 [1] : Starting with the JazzMIDIBridge example5 file (see [5]):
>     The original example5 file uses setTimeout() naively to simulate a
>     sequencer, and the first thing I wanted to do was implement the
>     approach described in [6] to cancel out the delays caused by other
>     processes running in the system. In other words, the delay passed
>     to setTimeout() is adjusted in this test so that
>            (systemTime + delay)  is equal to the MIDIMessage.timestamp.
>     Having done that, I set the code up to play a simple sequence of
>     50 NOTE_ON messages at 200 milliseconds per message.
>     The result is much more accurate than it would have been without
>     adjusting the delay.
>     I'm not too worried about Date().getTime()'s (lack of) accuracy.
>     Presumably the Web Audio timer would be better, but the principle
>     would not change. My machine has no problem playing this test
>     sequence completely accurately (according to the system clock).
>     Test 2 [2] : the same code as test 1, except that it tries to play
>     1000 NOTE_ON messages at 100 milliseconds per message. That should
>     take 100 seconds. As can be seen from the output, the delay value
>     being passed to setTimeout() tends to get smaller over time,
>     because the amount of time being used for other processes is
>     accumulating. Eventually, the delay is going to go negative, and
>     it will no longer be possible to honour the message's timestamp at
>     all. That can be seen happening, on my machine, in Test 3.
>     Test 3 [3]: is the same code as tests 1 & 2, but, with 2000
>     NOTE_ONs at 50 ms per message.
>     (I'm aware that there are issues connected with passing small
>     delay values to setTimeout().)
>     Test 4 [4]: This test uses 500 repeats of the sequence 50, 50, 50,
>     200 milliseconds.
>     I'm trying to play the sequence "as accurately as possible", and
>     tests 1-3 show that no computer on earth is going to be able to
>     play long, dense sequences accurately according to the timestamps
>     forever if it is simultaneously doing other things. (Is that right?)
>     So I've started using "time drift" or "local time": The base time
>     from which the timestamps are measured is allowed to drift, and
>     the drift occurs where it is least noticed -- during the longer
>     notes. Adding 10ms to a long duration is less noticeable than
>     adding it to a short one. I don't really care if a performance
>     which should last several minutes, is actually a few seconds longer.
>     So the function which uses the timestamp to calculate the delay
>     has to use the previous timestamp as well, in order to know
>     whether it can add a bit of drift or not. The previous timestamp
>     is therefore cached.
>     The algorithm for adding drift could probably be improved, but the
>     fact of having to add drift, and the algorithm's dependency on the
>     durations between timestamps is not going to change.
>     As defined, this sequence should should take exactly 175000
>     miliseconds (just short of 3 minutes), but the last time I ran it
>     it took about 4 seconds longer than that. But at least it sounded
>     okay (if you like that sort of thing) and got that far without
>     wanting to go backwards in time! :-)
>     Am I missing something? I'm not at all sure that this is
>     fundamentally the right way to go. I was rather hoping that there
>     could be accurate API functions for
>     playing/pausing/stopping/rewinding/ MIDI sequences. But that would
>     mean agreeing on Track and Sequence definitions. Which is where I
>     came in. :-)
>     all the best,
>     James
>     [1]
>     http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence1.html <http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
>     [2]
>     http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence2.html <http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
>     [3]
>     http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence3.html <http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
>     [4]
>     http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html
>     [5] https://github.com/abudaan/JazzMIDIBridge
>     [6] http://www.sitepoint.com/creating-accurate-timers-in-javascript/
>     Am 12.09.2012 01:14, schrieb Chris Wilson:
>         Hey James-
>         Actually, sendMIDIMessage takes MIDIMessages, which can have a
>         timestamp - it's not an "absolute" timestamp in the sense of
>         being relative to the UNIX epoch or anything, but you can in
>         fact send an entire buffer of MIDIMessages with their
>         timestamps to be sent at the same time.
>         This doesn't address wanting to change tempo dynamically (or
>         pausing), of course; but you absolutely can do this.  I'm not
>         sure it's what you'd want in most cases (other than simple
>         playback), but it is definitely there.
>         On Tue, Sep 11, 2012 at 2:00 AM, James Ingram
>         <j.ingram@netcologne.de <mailto:j.ingram@netcologne.de>
>         <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>>> wrote:
>             Hi Chris, Jussi, all,
>             Thanks, Chris, for this answer. I could certainly live
>         with using
>             a dedicated Javascript library which provided Tracks and
>             Sequences. Probably, some libraries will be more suitable
>         for some
>             tasks than others, and its best to leave that flexibility
>         outside
>             the MIDI API.
>             And I take your point that the current API needs to get locked
>             down before we start talking about v2 (libraries).
>             But I'm still wondering if the API is really complete as it
>             stands. I'm currently wrestling with the problem of accurate
>             timing, and am asking myself why this is such a hassle. If
>         there
>             was an accurate
>                  MIDIOutput.sendAtAbsoluteTime(midiIMessage, absoluteTime)
>             then the underlying code could take advantage of existing
>         timing
>             facilities, and the API's user (me) could get on with solving
>             other problems. I suspect that timing problems in
>         particular would
>             be best solved at a low level...
>             best,
>             James
>             Am 11.09.2012 00:48, schrieb Chris Wilson:
>                 Hey James -
>                 A while ago, I poked at implementing Standard MIDI File
>                 reading in Javascript
>                 (https://github.com/cwilso/Standard-MIDI-File-reader),
>         and my
>                 conclusion was basically that it would be best
>         provided as a
>                 library and/or as baked into the standard <audio> player
>                 engine in the browser, rather than centering the MIDI
>         API on
>                 it.  There are a lot of challenges with doing
>         rewinding, for
>                 example, and seeking.  There is also a lot of dynamic
>         control
>                 hooking in to tempo changes, live information exposure of
>                 events, etc., that any "serious" sequencer would
>         probably want
>                 to have.  Even in that relatively naive reader, there
>         are a
>                 bunch of different types of data - it has more
>         exposure than
>                 the MIDI API does by itself.  Once we've got the MIDI API
>                 locked down and implemented, I'll probably go back and
>         flesh
>                 out my SMF reader as a library for reading and playing
>         SMFs,
>                 and we can talk about if it's a useful v2 standard
>         library.
>                 -C
>                 On Mon, Sep 10, 2012 at 5:06 AM, James Ingram
>                 <j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de> <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>>
>                 <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>
>                 <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>>>> wrote:
>                     Hi Jussi,
>                     I think support for Sequences would be useful (and
>         easy to
>                     provide), even if we are not talking about file
>         output.
>                     Loading sequences from files is very common in MIDI
>                 programs, and
>                     I see no reason why that should not be done on the
>         web. So it
>                     would be nice to have a standard Sequence
>         definition with
>                 which to
>                     work. I'm not saying that Sequences necessarily
>         have to be
>                 loaded
>                     from SMFs, but if the Sequence definition existed,
>         it could
>                     obviously be given a loadFromSMF(url) member function.
>                     Also, if we had a Sequence object, it would wrap
>         the output
>                     device, and play(), stop(), rewind() etc.
>         functions could be
>                     provided quite cheaply out of the box. That might
>         make Web
>                 MIDI
>                     even more attractive. :-)
>                     But maybe I'm too close to the problem. Is there
>         anyone
>                 else out
>                     there who needs to send a Sequence to the output
>         device?
>                     best,
>                     James
>                         Hi James,
>                         A side note: the current spec has the notion of
>                     MIDIEvent as a
>                         DOM event that occurs when MIDIMessage(s) are
>         received.
>                         The toolkit you're referring to deals both
>         with MIDI
>                     files (SMF)
>                         and MIDI devices. Our API is designed to work with
>                     just devices,
>                         and MIDI devices are just I/O, there's no
>         notion of
>                     high level
>                         concepts such as tracks (although there's
>         channels,
>                     but that's
>                         already in the spec) or sequences, unlike SMF
>         does. It
>                     would seem
>                         a bit weird to me to complicate the spec with such
>                     high-level
>                         concepts as we don't even support reading /
>         writing
>                     SMF files,
>                         nor do I think we should as the intent of the
>         device is
>                         explicitly to allow communication with MIDI
>         devices.
>                         Cheers,
>                         Jussi
>                         On Mon, Sep 10, 2012 at 12:32 PM, James Ingram
>                         <j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>
>                     <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>>
>                     <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>
>                     <mailto:j.ingram@netcologne.de
>         <mailto:j.ingram@netcologne.de>>>> wrote:
>                             Hi Chris, all,
>                             Chris Wilson said (apropos Bug 18764)
>                                 I'd like to explicitly ask for MORE
>         feedback
>                     from developers
>                                 using MIDI today on this issue.
>                             I'm following the Bug 18764 debate very
>         closely,
>                     but don't
>                             have the in-depth knowledge of MIDI's
>         finer points to
>                             contribute much to the subject.  All I
>         want as a
>                     future user
>                             of this API is that it is going to be
>         succinct,
>                     work as
>                             efficiently as possible and be clearly
>         documented.
>                     Whether
>                             functions are overloaded or not is not really
>                     crucial. If I
>                             want to use the Web MIDI API, then I'm
>         going to
>                     have to learn
>                             it one way or the other.
>                             But I'd like to open another can of worms: :-)
>                             The current version of the API [1] supports
>                     MIDIMessages and
>                             MIDIEvents. Would it be possible for it to
>         provide
>                     Sequence
>                             and Track objects as well? I've written my own
>                     versions of
>                             these for my current project (in
>         Javascript), but
>                     think there
>                             must be people out there who know much
>         better how
>                     to write
>                             them than I do. They are prime targets for
>                     optimization. We
>                             should not all have to re-invent the wheel.
>                             Interestingly, there are versions of these
>         objects
>                     in Leslie
>                             Sanford's popular C# MIDI Toolkit [2]., so
>         he also
>                     seems to
>                             have thought that they were worth providing.
>                             Any thoughts?
>                             Best,
>                             James
>                             [1]
>         https://dvcs.w3.org/hg/audio/raw-file/tip/midi/specification.html#callback-navigatormidiaccesssuccesscallback-parameters
>                             [2]
>         http://www.codeproject.com/Articles/6228/C-MIDI-Toolkit
>                     -- www.james-ingram-act-two.de
>         <http://www.james-ingram-act-two.de>
>                 <http://www.james-ingram-act-two.de>
>                  <http://www.james-ingram-act-two.de>
>             -- www.james-ingram-act-two.de
>         <http://www.james-ingram-act-two.de>
>         <http://www.james-ingram-act-two.de>
>     -- 
>     www.james-ingram-act-two.de <http://www.james-ingram-act-two.de>

Received on Wednesday, 12 September 2012 21:59:03 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 21:50:02 UTC