Re: A working MusicXML engraver?

Trying to define a 'correct' MusicXML renderization could be difficult or
even impossible, specially taking into account that positioning information
is not mandatory, and that many renderization models (fixed page size,
continuous flow  as in web pages, etc.) are possible. Therefore, I think
that a 'reference' MusicXML rendered is not important and trying to have it
could deviate a lot of time and resources.

But I strongly support the idea, commented by others, that it is more
important to have a good test set that covers as much as possible of use
cases. Ideally, each test file should be small and trying to cover just one
test case. The whole test suite could serve also as reference for the
"correct" way of encoding music notation, specially in those cases in which
alternatives are possible.

Regards,
Cecilio

2015-12-02 11:06 GMT+01:00 Andreas Wenger <andi.xenoage@googlemail.com>:

> As we've discussed, screenshot-based testing is probably not a good
> solution for a general MusicXML engraving test suite.
> So how could a better test case look like?
>
> Let's come back to L Peter Deutsch's original problem. For example, he
> mentioned:
>
> Sibelius 6.1 [...] misinterpreted a parenthesized accidental (omitted the parens
>   and moved it to a different note).
>
>
> We could define a test case like this one from the Unofficial MusicXML
> test suite:
> http://www.lilypond.org/doc/v2.18/input/regression/musicxml/17/lily-3f3527a9.png
>
> Assertions for this test case:
> 4 Measures, each with 4 quarter notes. For each measure: Note 1 has an
> accidental attached to it. It must be on the same vertical position and
> position near to the left side of the note. Note 2 has no accidental
> attached. Note 3 and 4 has an accidental with parenthesis attached. It must
> be on the same vertical position and a little more to the left than the
> accidental on note 1.
>
> How can we assert this?
> In Zong! we have data structures for each level of processing. First, we
> hold the MusicXML structure in memory, which we can already test (correct
> number of measures/notes? parenthesis on the right accidentals?). Then, we
> have our internal score representation, which we can also test (again the
> measures, notes and accidentals). After layouting, we have the graphical
> representation of the score objects (knowing their musical context), so we
> can easily perform the upper test. The final rendering to an image is then
> a trivial process which needs not to be tested.
>
> I do not know enough about the way other MusicXML engravers work. Maybe it
> is not possible for some to get the intermediate results on the way from
> MusicXML to the final image. But when it is possible, assertions like the
> above ones seem definitely more helpful to me than a pixelwise screenshot
> comparison.
>
> Bye, Andi
>
>
>
>
>
> Am 01.12.2015 um 20:30 schrieb Joe Berkovitz:
>
> I'm going to speak to what I imagine our needs may be down the road a ways
> -- I understand that this thread is also sharing some information about
> various renderers of today, and that's also very useful.
>
> It's very hard to create a reasonable quality notation renderer or editor
> without a full test suite. Noteflight, like other products/projects in this
> area, has an extensive set of visual engraving regression tests. However, I
> am not sure an screenshot-diff approach like Noteflight's or Soundslice's
> is going to address the nature of the tests needed to check correctness in
> a future MusicXML specification. Consider what we will want to assert. We
> are likely to be making assertions about positional relationships between
> glyphs based on their registration points, with considerable flexibility as
> to other details of layout that are not prescribed by the future spec (and
> which are irrelevant to the test results).
>
> To put this another way: I suspect a MusicXML test suite should not test
> the details of fully realized, high-quality music engraving. It is probably
> going to be a test of whatever MusicXML requires that a renderer do, and no
> more:  the small subset of accepted engraving rules and practices that can
> be safely prescribed by a future spec. Unless we want an endless and
> ultimately unanswerable series of debates on said rules and practices...
>
> ...Joe
>
>
> .            .       .    .  . ...Joe
>
> *Joe Berkovitz*
> President
>
> *Noteflight LLC*
> 49R Day Street / Somerville, MA 02144 / USA
> phone: +1 978 314 6271
> <http://www.noteflight.com>www.noteflight.com
> "Your music, everywhere"
>
> On Tue, Dec 1, 2015 at 1:32 PM, Joshan Mahmud <joshan.mahmud@gmail.com>
> wrote:
>
>> That would be very useful Adrian.  VexFlow similarly has a unit test
>> suite which tests all functionality of it's codebase and essentially has a
>> single page where a set of small unit-based bits of engraving are
>> demonstrated.  In my project I am essentially doing the same, but instead
>> the 'Arrange' part of my test starts with some MusicXML, and the outcome /
>> 'Assert' is the rendered music.  It would be nice to have this sort of set
>> up for each part of the specification as it shows a practical example which
>> can be run (and debugged).  Having this along with screenshots of the
>> original would be really really useful!  (Again this is what would be
>> useful to me, and not sure if it would suit everyone).
>>
>> Thanks
>> Josh
>>
>> On Tue, Dec 1, 2015 at 5:17 PM, Adrian Holovaty < <adrian@holovaty.com>
>> adrian@holovaty.com> wrote:
>>
>>> On Tue, Dec 1, 2015 at 1:55 PM, Andreas Wenger <
>>> andi.xenoage@googlemail.com> wrote:
>>>
>>>> But a test suite in this style (MusicXML input and expected result
>>>> image) seems to be perfect for me to measure quality. What tests/benchmarks
>>>> do the other MusicXML engravers use?
>>>>
>>>
>>> For Soundslice, we have an extensive internal test suite that tests for
>>> regressions each time we improve the engraving engine. It takes screenshots
>>> and compares them to the baseline (correct) versions. I wrote about the
>>> infrastructure here:
>>> http://www.holovaty.com/writing/automated-screenshot-tests/
>>>
>>> I would be happy to make a freely available repository of MusicXML
>>> tests, rendered with our engine. Sure, we're proprietary, but our rendering
>>> is pretty solid and the only proprietary bit would be the rendering engine,
>>> anyway -- not the screenshots or MusicXML files themselves.
>>>
>>> Adrian
>>>
>>
>>
>
>