Re: A working MusicXML engraver?

As we've discussed, screenshot-based testing is probably not a good 
solution for a general MusicXML engraving test suite.
So how could a better test case look like?

Let's come back to L Peter Deutsch's original problem. For example, he 
mentioned:

> Sibelius 6.1 [...] misinterpreted a parenthesized accidental (omitted the parens
>    and moved it to a different note).

We could define a test case like this one from the Unofficial MusicXML 
test suite: 
http://www.lilypond.org/doc/v2.18/input/regression/musicxml/17/lily-3f3527a9.png

Assertions for this test case:
4 Measures, each with 4 quarter notes. For each measure: Note 1 has an 
accidental attached to it. It must be on the same vertical position and 
position near to the left side of the note. Note 2 has no accidental 
attached. Note 3 and 4 has an accidental with parenthesis attached. It 
must be on the same vertical position and a little more to the left than 
the accidental on note 1.

How can we assert this?
In Zong! we have data structures for each level of processing. First, we 
hold the MusicXML structure in memory, which we can already test 
(correct number of measures/notes? parenthesis on the right 
accidentals?). Then, we have our internal score representation, which we 
can also test (again the measures, notes and accidentals). After 
layouting, we have the graphical representation of the score objects 
(knowing their musical context), so we can easily perform the upper 
test. The final rendering to an image is then a trivial process which 
needs not to be tested.

I do not know enough about the way other MusicXML engravers work. Maybe 
it is not possible for some to get the intermediate results on the way 
from MusicXML to the final image. But when it is possible, assertions 
like the above ones seem definitely more helpful to me than a pixelwise 
screenshot comparison.

Bye, Andi




Am 01.12.2015 um 20:30 schrieb Joe Berkovitz:
> I'm going to speak to what I imagine our needs may be down the road a 
> ways -- I understand that this thread is also sharing some information 
> about various renderers of today, and that's also very useful.
>
> It's very hard to create a reasonable quality notation renderer or 
> editor without a full test suite. Noteflight, like other 
> products/projects in this area, has an extensive set of visual 
> engraving regression tests. However, I am not sure an screenshot-diff 
> approach like Noteflight's or Soundslice's is going to address the 
> nature of the tests needed to check correctness in a future MusicXML 
> specification. Consider what we will want to assert. We are likely to 
> be making assertions about positional relationships between glyphs 
> based on their registration points, with considerable flexibility as 
> to other details of layout that are not prescribed by the future spec 
> (and which are irrelevant to the test results).
>
> To put this another way: I suspect a MusicXML test suite should not 
> test the details of fully realized, high-quality music engraving. It 
> is probably going to be a test of whatever MusicXML requires that a 
> renderer do, and no more:  the small subset of accepted engraving 
> rules and practices that can be safely prescribed by a future spec. 
> Unless we want an endless and ultimately unanswerable series of 
> debates on said rules and practices...
>
> ...Joe
>
>
> .            .       .    .  . ...Joe
>
> *Joe Berkovitz*
> President
>
> *Noteflight LLC*
> 49R Day Street / Somerville, MA 02144 / USA
> phone: +1 978 314 6271
> www.noteflight.com <http://www.noteflight.com>
> "Your music, everywhere"
>
> On Tue, Dec 1, 2015 at 1:32 PM, Joshan Mahmud <joshan.mahmud@gmail.com 
> <mailto:joshan.mahmud@gmail.com>> wrote:
>
>     That would be very useful Adrian.  VexFlow similarly has a unit
>     test suite which tests all functionality of it's codebase and
>     essentially has a single page where a set of small unit-based bits
>     of engraving are demonstrated.  In my project I am essentially
>     doing the same, but instead the 'Arrange' part of my test starts
>     with some MusicXML, and the outcome / 'Assert' is the rendered
>     music.  It would be nice to have this sort of set up for each part
>     of the specification as it shows a practical example which can be
>     run (and debugged).  Having this along with screenshots of the
>     original would be really really useful!  (Again this is what would
>     be useful to me, and not sure if it would suit everyone).
>
>     Thanks
>     Josh
>
>     On Tue, Dec 1, 2015 at 5:17 PM, Adrian Holovaty
>     <adrian@holovaty.com <mailto:adrian@holovaty.com>> wrote:
>
>         On Tue, Dec 1, 2015 at 1:55 PM, Andreas Wenger
>         <andi.xenoage@googlemail.com
>         <mailto:andi.xenoage@googlemail.com>> wrote:
>
>             But a test suite in this style (MusicXML input and
>             expected result image) seems to be perfect for me to
>             measure quality. What tests/benchmarks do the other
>             MusicXML engravers use?
>
>
>         For Soundslice, we have an extensive internal test suite that
>         tests for regressions each time we improve the engraving
>         engine. It takes screenshots and compares them to the baseline
>         (correct) versions. I wrote about the infrastructure here:
>         http://www.holovaty.com/writing/automated-screenshot-tests/
>
>         I would be happy to make a freely available repository of
>         MusicXML tests, rendered with our engine. Sure, we're
>         proprietary, but our rendering is pretty solid and the only
>         proprietary bit would be the rendering engine, anyway -- not
>         the screenshots or MusicXML files themselves.
>
>         Adrian
>
>
>