Re: A working MusicXML engraver?

Joe,
I think that's extremely reasonable. We don't want to fall into a
bottomless pit of edge cases. I don't know what Noteflight's or
Soundslice's test suites look like, but I think the Vexflow test suite is
both relatively small while covering a large range of use cases. I think
MusicXML should strive toward a similar philosophy of creating a small test
suite that covers a lot of primary use cases.

However, there is also the question of where SMuFL fits into these test
cases, if at all (I think it should). As an example, SMuFL 1.18, p. 43ff
discusses bounding box cutouts. Whether the engraver uses these cutouts or
traditional bounding boxes will create different results. This will only
complicate any test cases, even if we require the test cases to use a
specific SMuFL-compatible font.

On Tue, Dec 1, 2015 at 2:30 PM, Joe Berkovitz <joe@noteflight.com> wrote:

> I'm going to speak to what I imagine our needs may be down the road a ways
> -- I understand that this thread is also sharing some information about
> various renderers of today, and that's also very useful.
>
> It's very hard to create a reasonable quality notation renderer or editor
> without a full test suite. Noteflight, like other products/projects in this
> area, has an extensive set of visual engraving regression tests. However, I
> am not sure an screenshot-diff approach like Noteflight's or Soundslice's
> is going to address the nature of the tests needed to check correctness in
> a future MusicXML specification. Consider what we will want to assert. We
> are likely to be making assertions about positional relationships between
> glyphs based on their registration points, with considerable flexibility as
> to other details of layout that are not prescribed by the future spec (and
> which are irrelevant to the test results).
>
> To put this another way: I suspect a MusicXML test suite should not test
> the details of fully realized, high-quality music engraving. It is probably
> going to be a test of whatever MusicXML requires that a renderer do, and no
> more:  the small subset of accepted engraving rules and practices that can
> be safely prescribed by a future spec. Unless we want an endless and
> ultimately unanswerable series of debates on said rules and practices...
>
> ...Joe
>
>
> .            .       .    .  . ...Joe
>
> *Joe Berkovitz*
> President
>
> *Noteflight LLC*
> 49R Day Street / Somerville, MA 02144 / USA
> phone: +1 978 314 6271
> www.noteflight.com
> "Your music, everywhere"
>
> On Tue, Dec 1, 2015 at 1:32 PM, Joshan Mahmud <joshan.mahmud@gmail.com>
> wrote:
>
>> That would be very useful Adrian.  VexFlow similarly has a unit test
>> suite which tests all functionality of it's codebase and essentially has a
>> single page where a set of small unit-based bits of engraving are
>> demonstrated.  In my project I am essentially doing the same, but instead
>> the 'Arrange' part of my test starts with some MusicXML, and the outcome /
>> 'Assert' is the rendered music.  It would be nice to have this sort of set
>> up for each part of the specification as it shows a practical example which
>> can be run (and debugged).  Having this along with screenshots of the
>> original would be really really useful!  (Again this is what would be
>> useful to me, and not sure if it would suit everyone).
>>
>> Thanks
>> Josh
>>
>> On Tue, Dec 1, 2015 at 5:17 PM, Adrian Holovaty <adrian@holovaty.com>
>> wrote:
>>
>>> On Tue, Dec 1, 2015 at 1:55 PM, Andreas Wenger <
>>> andi.xenoage@googlemail.com> wrote:
>>>
>>>> But a test suite in this style (MusicXML input and expected result
>>>> image) seems to be perfect for me to measure quality. What tests/benchmarks
>>>> do the other MusicXML engravers use?
>>>>
>>>
>>> For Soundslice, we have an extensive internal test suite that tests for
>>> regressions each time we improve the engraving engine. It takes screenshots
>>> and compares them to the baseline (correct) versions. I wrote about the
>>> infrastructure here:
>>> http://www.holovaty.com/writing/automated-screenshot-tests/
>>>
>>> I would be happy to make a freely available repository of MusicXML
>>> tests, rendered with our engine. Sure, we're proprietary, but our rendering
>>> is pretty solid and the only proprietary bit would be the rendering engine,
>>> anyway -- not the screenshots or MusicXML files themselves.
>>>
>>> Adrian
>>>
>>
>>
>