W3C home > Mailing lists > Public > public-audio@w3.org > April to June 2012

Re: Audio-ISSUE-105 (MIDI timestamp resolution): timestamps in MIDI should use High Resolution Time [MIDI API]

From: Adam Goode <agoode@google.com>
Date: Mon, 4 Jun 2012 21:17:37 -0400
Message-ID: <CAOf41NkE0Hg0+Gy2OKRoQahsWr4BLSAgCp5WO8ZqdpCgNUfaNw@mail.gmail.com>
To: Chris Wilson <cwilso@google.com>
Cc: Jussi Kalliokoski <jussi.kalliokoski@gmail.com>, public-audio@w3.org
On Mon, Jun 4, 2012 at 1:41 PM, Chris Wilson <cwilso@google.com> wrote:
> I had some further discussion off-list with Chris, where he helped me
> coalesce my thinking a little better.
>
> I think there are three design factors to a timestamp-
>
> What type the actual numerical value is (e.g. WebAudio's currentTime/time
> params are double, as is DOMHighResTimeStamp - some others are long ints, or
> long longs in Windows).  More importantly for JS devs, though is what the
> numeric value represents: i.e., number of seconds (in WebAudio) or number of
> milliseconds (DOMHRTS and MIDI proposal).

I think it should be milliseconds, this is consistent with the Date
constructor which has been around in JavaScript for a long time:
https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Date

Extending it to a double makes it easy to increase the resolution
beyond milliseconds.

> What the "zero reference" (or start point) is - e.g. in WA, it's when the
> audioContext is created, for DOMHRTS in the window.performance.now() usage
> it's navigationStart (although by itself, the DOMHRTS type does not imply a
> "start point" - we need to fix this if we want to use it in the MIDI API,
> even if it's just to say it's the same as window.performance.now().)  For
> Date.now(), the zero reference/start point is 1 January 1970 00:00:00

This is an important issue. For MIDI, I see an important use case
being input to rhythm or other kinds of games. So it should have
timestamps that can be easily consumed by a game engine. Should this
be unified with requestAnimationFrame (which is switching to DOMHRTS)?
Probably, since that seems like a reasonable game engine input. Also
worth considering are the timestamps used by Gamepad:
http://www.w3.org/TR/gamepad/#idl-def-Gamepad
But Gamepad gives no zero-time definition.

> On what clock the time proceeds.  This is somewhat subtle, because this is
> really to account for different clocks possibly running on different
> crystals - e.g., for WA this is the audio hardware clock. Like Data.now(),
> the performance.now() clock is just the system clock (except it's explicitly
> not supposed to adjust for system time[1]).  But just because you're using
> DOMHRTS type doesn't explicitly mean you're using the system clock -
> sections 4.3 and 4.4 are specific to the usage in window.performance.now(),
> and the rest of the spec simply defines the type (= #1 above).
>

Does WA distinguish between DATA and CONTROL? If so, then I see the
DATA part (audio samples) being driven by the audio hardware's clock,
but the CONTROL (noteOn, noteOff, etc, and MIDI input/output?)
controlled by the system clock. But I am newly studying all this
stuff, so I might be way off here.

> For the purposes of the MIDI API, I don't have any real problem with using
> the DOMHRTS type - but I'm a) not really psyched about having to use
> window.performance.now() in a MIDI application to get the zero-reference,
> because it seems like that is really designed to be a performance-timing
> thing, and b) a little less pleased that Web Audio times are in seconds, and
> this timestamp would be in milliseconds.  Neither of those are completely
> disastrous, but they both seem less than optimal.  Anyone else have thoughts
> on that?  Any thoughts about how developers usually think?

I would like to see Web Audio switch to using milliseconds, since
there is JavaScript and DOMHRTS precedent.

>
> -C
>
> [1]  http://www.w3.org/TR/2012/CR-hr-time-20120522/#sec-monotonic-clock.
>  This makes complete sense in the performance context - you wouldn't want
> performance measurements to be thrown by Daylight Savings time adjustments
> or the user or network time adjust changing the current time.  That would
> also make sense in a MIDI sequencer, so I guess mostly my objection a) boils
> down to "I think performance is the wrong place to put this," which is
> beyond our scope.  Sigh.
>
>
> On Fri, Jun 1, 2012 at 1:34 PM, Chris Wilson <cwilso@google.com> wrote:
>>
>> This was what was making me uncomfortable - it's defined as relative to
>> the Performance interface, navigation start time, which I think is a bit
>> odd.
>>
>>
>> On Fri, Jun 1, 2012 at 1:12 PM, Jussi Kalliokoski
>> <jussi.kalliokoski@gmail.com> wrote:
>>>
>>> Ah, you're right, in that case I don't see a reason to duplicate it.
>>>
>>> Cheers,
>>> Jussi
>>>
>>> On Jun 1, 2012 11:09 PM, "Adam Goode" <agoode@google.com> wrote:
>>> >
>>> > On Fri, Jun 1, 2012 at 4:02 PM, Jussi Kalliokoski
>>> > <jussi.kalliokoski@gmail.com> wrote:
>>> > > The reason I kind of like the idea of having the timestamps specified
>>> > > as
>>> > > DOMHighResTimeStamps is that it will allow the accuracy to live
>>> > > outside the
>>> > > spec, for example if in the future it somehow becomes desirable to
>>> > > have more
>>> > > accuracy than double precision, the DOMHighResTimeStamp will probably
>>> > > be
>>> > > updated by then to use a higher precision as well. Although I don't
>>> > > think a
>>> > > use case for higher resolution than double will come along very soon.
>>> > > Having
>>> > > the timestamps be related to the creation time of the MIDIAccess is a
>>> > > very
>>> > > good idea actually, because it makes the problem of accuracy
>>> > > deterioration a
>>> > > slightly smaller problem. We probably need to introduce some method
>>> > > to get
>>> > > the current timestamp of the MIDIAccess as well.
>>> > >
>>> >
>>> > If I'm reading the spec correctly, DOMHighResTimeStamp is defined to
>>> > be relative to the start of navigation to the page. I don't think we
>>> > should redefine this, but we could store the creation timestamp into
>>> > each MIDIAccess object if necessary.
>>> >
>>> > http://www.w3.org/TR/2012/CR-hr-time-20120522/#sec-DOMHighResTimeStamp
>>> >
>>> >
>>> > Adam
>>> >
>>> >
>>> > > Cheers,
>>> > > Jussi
>>> > >
>>> > >
>>> > > On Fri, Jun 1, 2012 at 10:39 PM, Chris Wilson <cwilso@google.com>
>>> > > wrote:
>>> > >>
>>> > >> Gah!  yes, sorry, didn't hit reply-all. Only thing in Gmail I'm
>>> > >> still not
>>> > >> quite used to, somehow.
>>> > >>
>>> > >> Yes, I agree that it's not great to have so many different timestamp
>>> > >> formats and reference points.  If the desire is to divorce from
>>> > >> wallclock
>>> > >> time, then I supposed we could do like audioContext does - from when
>>> > >> MIDIAccess is created.  As written in Jussi's last edit, though,
>>> > >> it's
>>> > >> "current time" (unfortunately, the definition of what that means (ms
>>> > >> since
>>> > >> UNIX epoch) was removed).  I don't have strong feelings.  I mostly
>>> > >> disliked
>>> > >> DOMHighResTimeStamp because it's one more reference, for what is
>>> > >> essentially
>>> > >> a trivial thing (monotonically increasing, number of milliseconds,
>>> > >> unrelated
>>> > >> to wallclock time), but that spec is really defined for uses
>>> > >> relating to
>>> > >> Performance, so it's confusing to read as a solution for this
>>> > >> problem.  I
>>> > >> think we would need to define our own zero point.
>>> > >>
>>> > >> I like seconds just because I think if it's not integer anyway, it's
>>> > >> easier for humans to think that way, but I don't care that strongly.
>>> > >>  The
>>> > >> newer MIDI interfaces in Windows, I note, use a longlong (64bit int)
>>> > >> of
>>> > >> units of 100ns (i.e. tenths of a microsecond, or 0.0001
>>> > >> milliseconds).  I
>>> > >> think that is kind of confusing, personally.  Seconds are prevalent
>>> > >> in the
>>> > >> Web Audio API, but milliseconds (as ints) are common in other web
>>> > >> programming APIs, so I could be okay with either.
>>> > >>
>>> > >> On Fri, Jun 1, 2012 at 12:17 PM, Adam Goode <agoode@google.com>
>>> > >> wrote:
>>> > >>>
>>> > >>> On Fri Jun 01 13:53:52 GMT-400 2012, Chris Wilson
>>> > >>> <cwilso@google.com>
>>> > >>> wrote:
>>> > >>>>
>>> > >>>> Well there you go - it's been quite a while since I wrote Windows
>>> > >>>> code.
>>> > >>>>  :)
>>> > >>>>
>>> > >>>>
>>> > >>>> >The point of DOMHighResTimeStamp is that it is divorced from
>>> > >>>> > wallclock time.
>>> > >>>>
>>> > >>>> So is audioContext.currentTime.
>>> > >>>>
>>> > >>>
>>> > >>> Hmmm. It's not great to have so many different timestamp formats
>>> > >>> and
>>> > >>> reference points. It does make sense for audioContext to have its 0
>>> > >>> point at
>>> > >>> its start time. And there is no "start time" for these raw MIDI
>>> > >>> events. So
>>> > >>> deferring to page load time seems fine.
>>> > >>>
>>> > >>> But the units are different (seconds in float vs. milliseconds in
>>> > >>> double), and that seems worth addressing.
>>> > >>>
>>> > >>>
>>> > >>> (Did we drop off the public list with this thread?)
>>> > >>>
>>> > >>> Adam
>>> > >>>
>>> > >>>> On Fri, Jun 1, 2012 at 10:49 AM, Adam Goode <agoode@google.com>
>>> > >>>> wrote:
>>> > >>>>
>>> > >>>> On Fri, Jun 1, 2012 at 12:47 PM, Chris Wilson <cwilso@google.com>
>>> > >>>> wrote:
>>> > >>>> >
>>> > >>>> > Although I'm not completely opposed to this change, I'd argue
>>> > >>>> > against
>>> > >>>> > the point that millisecond resolution is insufficient.  If using
>>> > >>>> > hardware
>>> > >>>> > MIDI ports, it takes approximately 1/4 of a millisecond to SEND
>>> > >>>> > a single
>>> > >>>> > byte of data - so it will take approximately 3/4 of a
>>> > >>>> > millisecond to simply
>>> > >>>> > transfer the data anyway - and the latency in processing at the
>>> > >>>> > other end is
>>> > >>>> > typically much, much higher than 1ms (I seem to recall around
>>> > >>>> > 4-7ms was not
>>> > >>>> > atypical for hardware synths, but can't find my reference ATM).
>>> > >>>> >
>>> > >>>>
>>> > >>>> The issue is more of jitter, not of processing delay. Though 1ms
>>> > >>>> seems
>>> > >>>> totally sufficient to me, I could imagine issues with the single
>>> > >>>> byte
>>> > >>>> timing code (F8) getting some unwanted jitter. But the real win of
>>> > >>>> this change is monotonicity.
>>> > >>>>
>>> > >>>> >
>>> > >>>> > That said, of course, it's not a bad idea to future-proof better
>>> > >>>> > than
>>> > >>>> > that; many MIDI use cases will never actually see a 5-pin-DIN
>>> > >>>> > cable.
>>> > >>>> >  However,
>>> > >>>> >
>>> > >>>> > 1) I find the usage of DOMHighResTimeStamp very confusing, as
>>> > >>>> > it's
>>> > >>>> > deliberately chained to (in terms of "zero" point) to the
>>> > >>>> > Performance
>>> > >>>> > interface.  It doesn't seem to add any value to reference here,
>>> > >>>> > since it's
>>> > >>>> > simply a double; we would still need to provide a way to get
>>> > >>>> > system time in
>>> > >>>> > double units, as I don't think using the PerformanceTiming
>>> > >>>> > interface is the
>>> > >>>> > most intuitive thing to do.  Or suggest that people use
>>> > >>>> > Date.now() (even
>>> > >>>> > though it's millisecond-precision), which is livable, I suppose.
>>> > >>>> >  But we do
>>> > >>>> > need to define that.  I would recommend either a) using a double
>>> > >>>> > for number
>>> > >>>> > of milliseconds, and recommending people use Date.now, or b) (my
>>> > >>>> > preference)
>>> > >>>> > use a double to represent number of seconds, to be uniform with
>>> > >>>> > the Web
>>> > >>>> > Audio API.  I'm ambivalent about whether we use the same
>>> > >>>> > currentTime from
>>> > >>>> > the audioContext as WA or Date.now().
>>> > >>>> >
>>> > >>>>
>>> > >>>> The point of DOMHighResTimeStamp is that it is divorced from
>>> > >>>> wallclock
>>> > >>>> time. All the MIDI implementations use this kind of time stamp
>>> > >>>> (even
>>> > >>>> Windows, read on).
>>> > >>>>
>>> > >>>>
>>> > >>>> >
>>> > >>>> > 2) I would absolutely recommend that we (similar to
>>> > >>>> > DOMHighResTimeStamp) explicitly state that implementations are
>>> > >>>> > allowed to
>>> > >>>> > have millisecond-only precision in their implementation.  The
>>> > >>>> > underlying
>>> > >>>> > system APIs on Windows are based in milliseconds, for example -
>>> > >>>> > unless
>>> > >>>> > they're building another API, the time stamps on MIM_DATA are in
>>> > >>>> > milliseconds. The underlying API on OSX is a bit harder to
>>> > >>>> > determine
>>> > >>>> > precision, but I think it is higher.
>>> > >>>> >
>>> > >>>>
>>> > >>>> Actually the ONLY part of DirectMusic that is undeprecated (it
>>> > >>>> disappeared briefly in Vista, then was replaced in a service pack)
>>> > >>>> is
>>> > >>>> high resolution monotonic MIDI timestamps:
>>> > >>>>
>>> > >>>>
>>> > >>>> http://msdn.microsoft.com/en-us/library/ee416788(VS.85).aspx#ID4EFEAC
>>> > >>>> http://support.microsoft.com/kb/943253
>>> > >>>>
>>> > >>>>
>>> > >>>> So yes, we can specify that the timestamps might only have ms
>>> > >>>> resolution, but I don't think it's really required.
>>> > >>>>
>>> > >>>>
>>> > >>>> Adam
>>> > >>>>
>>> > >>>> >
>>> > >>>> >
>>> > >>>> >
>>> > >>>> > On Fri, Jun 1, 2012 at 8:48 AM, Jussi Kalliokoski
>>> > >>>> > <jussi.kalliokoski@gmail.com> wrote:
>>> > >>>> >>
>>> > >>>> >> This issue is now pending review per
>>> > >>>> >> https://dvcs.w3.org/hg/audio/rev/b78b7c5e906e .
>>> > >>>> >>
>>> > >>>> >>
>>> > >>>> >> On Fri, Jun 1, 2012 at 6:22 PM, Jussi Kalliokoski
>>> > >>>> >> <jussi.kalliokoski@gmail.com> wrote:
>>> > >>>> >>>
>>> > >>>> >>> Good catch, thank you! As I planned it, the timestamp should
>>> > >>>> >>> have
>>> > >>>> >>> been a floating point value, allowing for sub-millisecond
>>> > >>>> >>> precision, but
>>> > >>>> >>> actually DOMHighResTimeStamp is actually more fit fore this.
>>> > >>>> >>> I will make the necessary changes to the spec.
>>> > >>>> >>>
>>> > >>>> >>> Cheers,
>>> > >>>> >>> Jussi
>>> > >>>> >>>
>>> > >>>> >>>
>>> > >>>> >>> On Fri, Jun 1, 2012 at 6:16 PM, Audio Working Group Issue
>>> > >>>> >>> Tracker
>>> > >>>> >>> <sysbot+tracker@w3.org> wrote:
>>> > >>>> >>>>
>>> > >>>> >>>> Audio-ISSUE-105 (MIDI timestamp resolution): timestamps in
>>> > >>>> >>>> MIDI
>>> > >>>> >>>> should use High Resolution Time [MIDI API]
>>> > >>>> >>>>
>>> > >>>> >>>> http://www.w3.org/2011/audio/track/issues/105
>>> > >>>> >>>>
>>> > >>>> >>>> Raised by: Adam Goode
>>> > >>>> >>>> On product: MIDI API
>>> > >>>> >>>>
>>> > >>>> >>>> The current MIDI API specifies timestamp as a long
>>> > >>>> >>>> representing
>>> > >>>> >>>> "milliseconds from the UNIX Epoch".
>>> > >>>> >>>>
>>> > >>>> >>>> For MIDI applications, millisecond resolution is insufficient
>>> > >>>> >>>> and
>>> > >>>> >>>> can cause noticeable jitter.
>>> > >>>> >>>>
>>> > >>>> >>>> Using absolute wallclock time is also problematic, as it is
>>> > >>>> >>>> subject
>>> > >>>> >>>> to system clock skew.
>>> > >>>> >>>>
>>> > >>>> >>>> The MIDI timestamp should use High Resolution Time
>>> > >>>> >>>> (DOMHighResTimeStamp), which solves these problems:
>>> > >>>> >>>>
>>> > >>>> >>>>
>>> > >>>> >>>> http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HighResolutionTime/Overview.html
>>> > >>>> >>>>
>>> > >>>> >>>>
>>> > >>>> >>>>
>>> > >>>> >>>
>>> > >>>> >>
>>> > >>>> >
>>> > >>>>
>>> > >>>>
>>> > >>
>>> > >
>>
>>
>
Received on Tuesday, 5 June 2012 01:18:07 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 5 June 2012 01:18:12 GMT