Re: The ixml test suite

Norm Tovey-Walsh writes:

> I also think we should adopt Michael’s catalog format

I had hoped there would be other designs, so that we could compare them
and hybridize them before settling on one.

One thing has already become clear:  as long as we are working on the
spec and the test cases in parallel, we need a way to associate a test
case -- or rather, the expected result of a test case -- with a
particular version of the spec and the grammar.

> and organize the tests such that they aren’t under tests-SP-MSM.

If we are going to work on test sets as a group, we will also need to
add a mechanism for marking test cases (or expected results) as
disputed, or as reflecting a bug in the spec that has not yet been
resolved.

In the test suites for some specs I've worked on, different implementors
contributed test cases, and the simplest way to organize them was by
contributor, since that simply required throwing them in a directory
together with their catalog.  Making a more systematic structure would
require one-by-one evaluation of each test case.  The result is a nicer
organization of the tests, but the cost : benefit ratio seems
unattractive.

So I made the directory tests-SP-MSM just to hold the tests originally
contributed by SP and modified (corrected, it says in the catalog files)
by MSM.

However, I agree that the current state of affairs is confusing.  I just
don't have good ideas for a lowest-cost way of making it better.

Michael

-- 
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com

Received on Tuesday, 1 February 2022 15:16:14 UTC