Re: Graphical Scores and SVG

Hi Joe, Chris, Jeremy, Glenn etc.!

First, I'd like to say how glad I am that Joe has been talking to Chris 
Lilley, and that they have been looking at alternative strategies for 
using SVG in music notation. It would be really great if Chris could get 
involved at this early stage!

@Jeremy and @Glenn: Please accept my apologies for not replying to your 
points directly here. This posting is long enough already with my reply 
to Joe.

Joe said:

> Let me say first that I see at least two potential uses for SVG in our 
> community group, and they seem to harmonize perfectly:
>
> 1. SVG (with some relationship to sound) can represent musical scores 
> with arbitrary visual and sonic content. Thanks to James Ingram for 
> highlighting this particular case to the group.
I've only been talking about embedding MIDI data. The problem of 
synchronizing scores with audio files is covered in B and C below. 
Meanwhile, I'm defending my MIDI corner! :-)

Joe:
> 2. SVG (with some relationship to sound) can serve as an intermediate 
> format that represents *a specific rendering* of semantic music 
> notation markup such as MNX, MusicXML, MEI, or anything else.
I've partly replied to this (#2) today in the "Re: MNX Proposal now 
Available [via Music Notation Community Group]" thread. Here's a more 
technical idea:
Score rendering applications could convert MNX element names (e.g. 
<tuplet>) directly into class names in the exported SVG (<g 
class="tuplet" ...>...</g>).

That would mean that the SVG+MIDI client application could identify the 
class of a graphic object so as to use it meaningfully. For example, a 
<g class="staff" ...>... </g> could be manipulated graphically (e.g. 
turned a different colour, made transparent etc.) to indicate that it is 
disabled. That's true for all the graphical element types (e.g. "slur").

Joe:

> Relative to SVG, then, the key question is: What's the best way to 
> augment an SVG document with information on sonic performance? There 
> are multiple ways to do it. Chris and I discussed several very 
> high-level approaches:
>
> A. Intersperse performance data (e.g. individual MIDI or note events) 
> throughout an SVG file. James's proposed approach follows this 
> pattern: MIDI codes are sprinkled directly into the file and attached 
> to graphical elements. One could also use a different means to specify 
> events, say like the way that MNX proposes.
I think you are referring to §7 Interpretation.
Could it be that such CSS-type transformations should be available in 
both the MNX (the/input /to a score editor) and in the score editor's 
/output/ (SVG+MIDI)? As I pointed out in the other thread, temporal data 
can be manipulated, independently of the graphics, by an SVG+MIDI 
client, so it might be appropriate to put the CSS there as well. I'm not 
sure what interpretation data would mean as /input/ to a score editor... 
But it could be there anyway... The score editor could just pass the 
data on to its output.
My proposed <score:midi> element could indeed contain something more 
like your MIDI definitions. Do your MIDI definitions in any way restrict 
the kinds of MIDI message that can be sent? Should there be a hybrid 
version, in which arbitrary MIDI messages can be sent if necessary? 
That's very interesting. Needs thinking about.

Joe:
> B. Intersperse *references* to a separate performance file (e.g. a 
> Standard MIDI file, MP3 audio file) throughout an SVG file. In this 
> approach, SVG elements are tagged with simpler data that "points to" 
> some time or time range within a separate document. MEI uses this 
> technique in places.
> Example:
> <measure sound-ref="ThisPiece.midi" sound-start="1:22" 
> sound-end="1:24">...
As I understand it, this approach has its snags. There's a problem 
because the audio thread can't be synchronised exactly with the graphics 
display. (Joe, can you throw any light on that? I think there's a Web 
Audio threading issue.)
As I pointed out in the other thread, an SVG+MIDI client application can 
easily manipulate the playback (change speed arbitrarily etc.) while 
remaining synchronous. That would be very difficult to do if the score 
is linked to fixed recordings (SMFs or audio files). I think this 
approach (B) would be most useful (if the snags can be overcome) if one 
happens to have a particular audio file with which one wants to synchronize.

Joe:
> C. Create a completely separate mapping file that identifies a 
> correspondence between SVG and a performance file. Such a file might 
> contain entries like this:
> <measure-mapping graphics-ref="ThisPiece.svg#m21" 
> sound-ref="ThisPiece.midi" sound-start="1:22" sound-end="1:24"/>
Such an approach would mostly have the same advantages and disadvantages 
as B, so I have no real preference for one over the other. Why not allow 
them both?

Joe:
> [...] most of the good stuff in standards groups, happens on email and 
> Github over time, not in large in-person meetings!
Agreed. But in-person meetings are very important too! :-)

All the best,
James