Re: The ixml test suite

"C. M. Sperberg-McQueen" <cmsmcq@blackmesatech.com> writes:
> Norm Tovey-Walsh writes:
>
>> I also think we should adopt Michael’s catalog format
>
> I had hoped there would be other designs, so that we could compare them
> and hybridize them before settling on one.

What you’ve got seems fine. The catalog format is as amenable to pull
requests as anything else :-)

> One thing has already become clear:  as long as we are working on the
> spec and the test cases in parallel, we need a way to associate a test
> case -- or rather, the expected result of a test case -- with a
> particular version of the spec and the grammar.

I was going to ask about versioning the grammar but decided that would
open a can of worms and, frankly, there are too many such open cans at
the moment.

I think we might live with saying that as the spec evolves, so should
the tests. It’s a bit messy, but then so is a test that claims to pass
against 0.9.4a of the spec.

> If we are going to work on test sets as a group, we will also need to
> add a mechanism for marking test cases (or expected results) as
> disputed, or as reflecting a bug in the spec that has not yet been
> resolved.

Yep.

> In the test suites for some specs I've worked on, different implementors
> contributed test cases, and the simplest way to organize them was by
> contributor, since that simply required throwing them in a directory

I think that’s fine. I don’t mind directories by contributor, but
/steven/tests-SP-MSM has a weird ring about it.

Steven, I think what Michael has done is refactored your tests to fit
into the catalog. Are you happy to use that going forward?

> However, I agree that the current state of affairs is confusing.  I just
> don't have good ideas for a lowest-cost way of making it better.

If Steven agrees to support the catalog format, I’m happy to put
together a PR that reorganizes things so that /tests/steven contains
what’s currently in /tests/steven/tests-SP-MSM and we can go from there.

                                        Be seeing you,
                                          norm

--
Norm Tovey-Walsh
Saxonica