W3C home > Mailing lists > Public > public-audio@w3.org > July to September 2012

Re: MIDI Tracks and Sequences

From: Chris Wilson <cwilso@google.com>
Date: Wed, 12 Sep 2012 10:47:03 -0700
Message-ID: <CAJK2wqXUxY010GBNpVd7rH1nkQ+vtc+ZhbZdDqYFBNMz3jbEMg@mail.gmail.com>
To: James Ingram <j.ingram@netcologne.de>
Cc: Jussi Kalliokoski <jussi.kalliokoski@gmail.com>, public-audio@w3.org
I'm quite confused.  sendTimestampedMIDIMessage in that shim is not a
function you should have to write or edit; the Web MIDI API has a "send
this message, at this absolute time" capability.  You shouldn't need to
deal with clock drift; in a real, native implementation of the Web MIDI
API, a lower-level timing system would be handling this, not Javascript's
setTimeout/setInterval; much like the Web Audio API has its own system for
scheduling.  This is going to be a weak point in any polyfill (the one you
point to, and the one I'm writing) - you simply cannot get access to a
high-enough resolution clock to properly implement the timestamps, and none
of the current ways to access MIDI in Javascript that I know of expose a
higher-res timing system.

For playing/pausing/stopping, the only thing I think we may be missing is
the ability to cancel all scheduled MIDI messages, although before imposing
that I need to look more closely at how the underlying systems support such
a beast; you may need to close the device in order to clear it.

My suggested implementation tactic would be to use setInterval to
pre-schedule MIDI events in the send message queue, keeping some arbitrary
interval of time (a tenth of a second or so?) ahead of the current DOMHRT
time; this slower loop can then be cancelled whenever you hit stop.

On Wed, Sep 12, 2012 at 10:10 AM, James Ingram <j.ingram@netcologne.de>wrote:

> Hi Chris, Jussi, all,
> As you pointed out, this kind of thing may well belong in a standard
> library, but I think it might be worth taking a closer look at the problems
> that are going to have to be solved, in case we discover that lower level
> solutions might be better after all.
> As I said, I'm currently writing a sequencer in Javascript, using
> JazzMIDIBridge [5] to emulate the future Web MIDI API.
> I've created my own definition of a Sequence, and can now load such
> Sequences from files. The immediate problem is now to write a simple
>     sequence.play()
> function which will play the sequence, from beginning to end, as
> accurately as possible.
> In general, I expect sequences to be large (there should be no limit, in
> principle, to their duration), and to contain lots of continuous control
> data. So I'm creeping up on the problem, by writing a few test files. (I'm
> working on Windows, and using the latest Chrome browser.)
> Test 1 [1] : Starting with the JazzMIDIBridge example5 file (see [5]):
> The original example5 file uses setTimeout() naively to simulate a
> sequencer, and the first thing I wanted to do was implement the approach
> described in [6] to cancel out the delays caused by other processes running
> in the system. In other words, the delay passed to setTimeout() is adjusted
> in this test so that
>        (systemTime + delay)  is equal to the MIDIMessage.timestamp.
> Having done that, I set the code up to play a simple sequence of 50
> NOTE_ON messages at 200 milliseconds per message.
> The result is much more accurate than it would have been without adjusting
> the delay.
> I'm not too worried about Date().getTime()'s (lack of) accuracy.
> Presumably the Web Audio timer would be better, but the principle would not
> change. My machine has no problem playing this test sequence completely
> accurately (according to the system clock).
> Test 2 [2] : the same code as test 1, except that it tries to play 1000
> NOTE_ON messages at 100 milliseconds per message. That should take 100
> seconds. As can be seen from the output, the delay value being passed to
> setTimeout() tends to get smaller over time, because the amount of time
> being used for other processes is accumulating. Eventually, the delay is
> going to go negative, and it will no longer be possible to honour the
> message's timestamp at all. That can be seen happening, on my machine, in
> Test 3.
> Test 3 [3]: is the same code as tests 1 & 2, but, with 2000 NOTE_ONs at 50
> ms per message.
> (I'm aware that there are issues connected with passing small delay values
> to setTimeout().)
> Test 4 [4]: This test uses 500 repeats of the sequence 50, 50, 50, 200
> milliseconds.
> I'm trying to play the sequence "as accurately as possible", and tests 1-3
> show that no computer on earth is going to be able to play long, dense
> sequences accurately according to the timestamps forever if it is
> simultaneously doing other things. (Is that right?)
> So I've started using "time drift" or "local time": The base time from
> which the timestamps are measured is allowed to drift, and the drift occurs
> where it is least noticed -- during the longer notes. Adding 10ms to a long
> duration is less noticeable than adding it to a short one. I don't really
> care if a performance which should last several minutes, is actually a few
> seconds longer.
> So the function which uses the timestamp to calculate the delay has to use
> the previous timestamp as well, in order to know whether it can add a bit
> of drift or not. The previous timestamp is therefore cached.
> The algorithm for adding drift could probably be improved, but the fact of
> having to add drift, and the algorithm's dependency on the durations
> between timestamps is not going to change.
> As defined, this sequence should should take exactly 175000 miliseconds
> (just short of 3 minutes), but the last time I ran it it took about 4
> seconds longer than that. But at least it sounded okay (if you like that
> sort of thing) and got that far without wanting to go backwards in time! :-)
> Am I missing something? I'm not at all sure that this is fundamentally the
> right way to go. I was rather hoping that there could be accurate API
> functions for playing/pausing/stopping/**rewinding/ MIDI sequences. But
> that would mean agreeing on Track and Sequence definitions. Which is where
> I came in. :-)
> all the best,
> James
> [1] http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**
> sequence1.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence1.html><
> http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**sequence4.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
> >
> [2] http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**
> sequence2.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence2.html><
> http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**sequence4.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
> >
> [3] http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**
> sequence3.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence3.html><
> http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**sequence4.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
> >
> [4] http://james-ingram.de/tests/**JazzMIDIBridge/ji-known-**
> sequence4.html<http://james-ingram.de/tests/JazzMIDIBridge/ji-known-sequence4.html>
> [5] https://github.com/abudaan/**JazzMIDIBridge<https://github.com/abudaan/JazzMIDIBridge>
> [6] http://www.sitepoint.com/**creating-accurate-timers-in-**javascript/<http://www.sitepoint.com/creating-accurate-timers-in-javascript/>
> Am 12.09.2012 01:14, schrieb Chris Wilson:
>> Hey James-
>> Actually, sendMIDIMessage takes MIDIMessages, which can have a timestamp
>> - it's not an "absolute" timestamp in the sense of being relative to the
>> UNIX epoch or anything, but you can in fact send an entire buffer of
>> MIDIMessages with their timestamps to be sent at the same time.
>> This doesn't address wanting to change tempo dynamically (or pausing), of
>> course; but you absolutely can do this.  I'm not sure it's what you'd want
>> in most cases (other than simple playback), but it is definitely there.
>> On Tue, Sep 11, 2012 at 2:00 AM, James Ingram <j.ingram@netcologne.de<mailto:
>> j.ingram@netcologne.de**>> wrote:
>>     Hi Chris, Jussi, all,
>>     Thanks, Chris, for this answer. I could certainly live with using
>>     a dedicated Javascript library which provided Tracks and
>>     Sequences. Probably, some libraries will be more suitable for some
>>     tasks than others, and its best to leave that flexibility outside
>>     the MIDI API.
>>     And I take your point that the current API needs to get locked
>>     down before we start talking about v2 (libraries).
>>     But I'm still wondering if the API is really complete as it
>>     stands. I'm currently wrestling with the problem of accurate
>>     timing, and am asking myself why this is such a hassle. If there
>>     was an accurate
>>          MIDIOutput.sendAtAbsoluteTime(**midiIMessage, absoluteTime)
>>     then the underlying code could take advantage of existing timing
>>     facilities, and the API's user (me) could get on with solving
>>     other problems. I suspect that timing problems in particular would
>>     be best solved at a low level...
>>     best,
>>     James
>>     Am 11.09.2012 00:48, schrieb Chris Wilson:
>>         Hey James -
>>         A while ago, I poked at implementing Standard MIDI File
>>         reading in Javascript
>>         (https://github.com/cwilso/**Standard-MIDI-File-reader<https://github.com/cwilso/Standard-MIDI-File-reader>),
>> and my
>>         conclusion was basically that it would be best provided as a
>>         library and/or as baked into the standard <audio> player
>>         engine in the browser, rather than centering the MIDI API on
>>         it.  There are a lot of challenges with doing rewinding, for
>>         example, and seeking.  There is also a lot of dynamic control
>>         hooking in to tempo changes, live information exposure of
>>         events, etc., that any "serious" sequencer would probably want
>>         to have.  Even in that relatively naive reader, there are a
>>         bunch of different types of data - it has more exposure than
>>         the MIDI API does by itself.  Once we've got the MIDI API
>>         locked down and implemented, I'll probably go back and flesh
>>         out my SMF reader as a library for reading and playing SMFs,
>>         and we can talk about if it's a useful v2 standard library.
>>         -C
>>         On Mon, Sep 10, 2012 at 5:06 AM, James Ingram
>>         <j.ingram@netcologne.de <mailto:j.ingram@netcologne.de**>
>>         <mailto:j.ingram@netcologne.de
>>         <mailto:j.ingram@netcologne.de**>>> wrote:
>>             Hi Jussi,
>>             I think support for Sequences would be useful (and easy to
>>             provide), even if we are not talking about file output.
>>             Loading sequences from files is very common in MIDI
>>         programs, and
>>             I see no reason why that should not be done on the web. So it
>>             would be nice to have a standard Sequence definition with
>>         which to
>>             work. I'm not saying that Sequences necessarily have to be
>>         loaded
>>             from SMFs, but if the Sequence definition existed, it could
>>             obviously be given a loadFromSMF(url) member function.
>>             Also, if we had a Sequence object, it would wrap the output
>>             device, and play(), stop(), rewind() etc. functions could be
>>             provided quite cheaply out of the box. That might make Web
>>         MIDI
>>             even more attractive. :-)
>>             But maybe I'm too close to the problem. Is there anyone
>>         else out
>>             there who needs to send a Sequence to the output device?
>>             best,
>>             James
>>                 Hi James,
>>                 A side note: the current spec has the notion of
>>             MIDIEvent as a
>>                 DOM event that occurs when MIDIMessage(s) are received.
>>                 The toolkit you're referring to deals both with MIDI
>>             files (SMF)
>>                 and MIDI devices. Our API is designed to work with
>>             just devices,
>>                 and MIDI devices are just I/O, there's no notion of
>>             high level
>>                 concepts such as tracks (although there's channels,
>>             but that's
>>                 already in the spec) or sequences, unlike SMF does. It
>>             would seem
>>                 a bit weird to me to complicate the spec with such
>>             high-level
>>                 concepts as we don't even support reading / writing
>>             SMF files,
>>                 nor do I think we should as the intent of the device is
>>                 explicitly to allow communication with MIDI devices.
>>                 Cheers,
>>                 Jussi
>>                 On Mon, Sep 10, 2012 at 12:32 PM, James Ingram
>>                 <j.ingram@netcologne.de
>>             <mailto:j.ingram@netcologne.de**>
>>             <mailto:j.ingram@netcologne.de
>>             <mailto:j.ingram@netcologne.de**>>> wrote:
>>                     Hi Chris, all,
>>                     Chris Wilson said (apropos Bug 18764)
>>                         I'd like to explicitly ask for MORE feedback
>>             from developers
>>                         using MIDI today on this issue.
>>                     I'm following the Bug 18764 debate very closely,
>>             but don't
>>                     have the in-depth knowledge of MIDI's finer points to
>>                     contribute much to the subject.  All I want as a
>>             future user
>>                     of this API is that it is going to be succinct,
>>             work as
>>                     efficiently as possible and be clearly documented.
>>             Whether
>>                     functions are overloaded or not is not really
>>             crucial. If I
>>                     want to use the Web MIDI API, then I'm going to
>>             have to learn
>>                     it one way or the other.
>>                     But I'd like to open another can of worms: :-)
>>                     The current version of the API [1] supports
>>             MIDIMessages and
>>                     MIDIEvents. Would it be possible for it to provide
>>             Sequence
>>                     and Track objects as well? I've written my own
>>             versions of
>>                     these for my current project (in Javascript), but
>>             think there
>>                     must be people out there who know much better how
>>             to write
>>                     them than I do. They are prime targets for
>>             optimization. We
>>                     should not all have to re-invent the wheel.
>>                     Interestingly, there are versions of these objects
>>             in Leslie
>>                     Sanford's popular C# MIDI Toolkit [2]., so he also
>>             seems to
>>                     have thought that they were worth providing.
>>                     Any thoughts?
>>                     Best,
>>                     James
>>                     [1]
>>             https://dvcs.w3.org/hg/audio/**raw-file/tip/midi/**
>> specification.html#callback-**navigatormidiaccesssuccesscall**
>> back-parameters<https://dvcs.w3.org/hg/audio/raw-file/tip/midi/specification.html#callback-navigatormidiaccesssuccesscallback-parameters>
>>                     [2]
>>             http://www.codeproject.com/**Articles/6228/C-MIDI-Toolkit<http://www.codeproject.com/Articles/6228/C-MIDI-Toolkit>
>>             -- www.james-ingram-act-two.de
>>         <http://www.james-ingram-act-**two.de<http://www.james-ingram-act-two.de>
>> >
>>          <http://www.james-ingram-act-**two.de<http://www.james-ingram-act-two.de>
>> >
>>     --     www.james-ingram-act-two.de <http://www.james-ingram-act-**
>> two.de <http://www.james-ingram-act-two.de>>
> --
> www.james-ingram-act-two.de
Received on Wednesday, 12 September 2012 17:47:38 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 12 September 2012 17:47:41 GMT