Graphical Scores and SVG

Hi all,

I think this is a good moment to talk in more detail about the potential
uses of SVG in the work of this group since we've had a fair bit of
activity on that topic.

To that end, I've just had a very useful exchange with W3C's Chris Lilley,
who is copied on this email. Chris is a computer scientist who originally
chaired the W3C SVG Working Group when it began in 1998, and saw SVG
through all the way from an abstract idea to its modern realization. So
it's fair to say he's been observing and thinking about its uses for a very
long time, with an expert's perspective. Chris is also a member of the W3C
Web Audio Working Group which is responsible for both audio and MIDI
standards on the Web.

What I'll present here is my current thinking, informed by some thoughtful
points made by Chris -- who I'm encouraging to jump in with his own words.

Let me say first that I see at least two potential uses for SVG in our
community group, and they seem to harmonize perfectly:

1. SVG (with some relationship to sound) can represent musical scores with
arbitrary visual and sonic content. Thanks to James Ingram for highlighting
this particular case to the group.

2. SVG (with some relationship to sound) can serve as an intermediate
format that represents *a specific rendering* of semantic music notation
markup such as MNX, MusicXML, MEI, or anything else.

So far a lot of discussion has revolved around #1, but #2 is at least as
significant. Consider that it permits semantic markup to be converted into
a format that can be easily displayed and played back by an application
that is much simpler than one that processes semantic markup. Of course,
that format is necessarily quite limited in other ways (try transposing it
to another key, or reformatting it for a different display size!)  But, as
a final-delivery mechanism, #2 seems to have a lot of merit. It could
provide a standardized way to package the output of notation renderers,
regardless of what markup language they use. In fact, MathML (a semantic
markup language for math formulas) is routinely converted to SVG by various
standard libraries for exactly this purpose.

Now: I believe we don't need to get into a big debate about which use is
more important. They both are. Also, in neither case do they eliminate our
need for a semantic markup language within the confines of some known
musical idiom, so there's no need to stop that train from leaving the
station. MNX explicitly makes room for graphical encodings to be placed
within it.

Relative to SVG, then, the key question is: What's the best way to augment
an SVG document with information on sonic performance? There are multiple
ways to do it. Chris and I discussed several very high-level approaches:

A. Intersperse performance data (e.g. individual MIDI or note events)
throughout an SVG file. James's proposed approach follows this pattern:
MIDI codes are sprinkled directly into the file and attached to graphical
elements. One could also use a different means to specify events, say like
the way that MNX proposes.

B. Intersperse *references* to a separate performance file (e.g. a Standard
MIDI file, MP3 audio file) throughout an SVG file. In this approach, SVG
elements are tagged with simpler data that "points to" some time or time
range within a separate document. MEI uses this technique in places.
Example:
    <measure sound-ref="ThisPiece.midi" sound-start="1:22"
sound-end="1:24">...

C. Create a completely separate mapping file that identifies a
correspondence between SVG and a performance file. Such a file might
contain entries like this:
    <measure-mapping graphics-ref="ThisPiece.svg#m21"
sound-ref="ThisPiece.midi" sound-start="1:22" sound-end="1:24"/>

I do not think there is a clear winner among these, and I don't think we
should immediately get into the weeds. The next step in this discussion --
when we have it -- is to look at the pros and cons of these various
approaches for uniting graphical and sonic information. Each has advantages
and disadvantages, and they need to be brought to the surface in what will
be a lengthy discussion of its own. All of the above techniques have been
tried in different contexts and there are definite lessons to be learned.

As a corollary: let's stop debating the importance of pure graphical music
encoding. There is no need for a debate: we agree that it *is* important.
However, its role and its structure do need not to be settled in advance of
our work to move ahead on CWMN semantic music encoding. We will need time
to tease out the answers to the questions raised above.

Finally a word on Frankfurt: the co-chairs plan to devote a limited period
of time to discussing this topic, but it will certainly be smaller than
many would like (myself included). We are limited by the other big things
on the agenda. But, in truth, most of the good stuff in standards groups,
happens on email and Github over time, not in large in-person meetings!

Best,
.            .       .    .  . ...Joe

Joe Berkovitz
Founder
Noteflight LLC

49R Day Street
Somerville MA 02144
USA

"Bring music to life"
www.noteflight.com