Re: Refactor of the tests directory

Norm Tovey-Walsh writes:

> Hi folks,

> In the course of testing my implementation (now passing 171 of 173
> tests; one I think should pass and one I think requires discussion[1]),
> I reworked the tests directory completely:

>   https://github.com/invisibleXML/ixml/pull/32

Thank you.

Git question:  if I want to look at all the files directly, the simplest
way is to clone repo ndw:ixml and switch to branch test-suite, correct?
Or is there a better way?

> I hope that the changes are acceptable.

At a first glance, almost all of them look ok to me.

But I don't understand the changes to catalog-as-grammar-tests.xml. It
looks at first glance as if all the grammar-test elements have been
changed to test-case elements, e.g.


    <grammar-test>
      <result>
	<assert-not-a-grammar/>
      </result>
    </grammar-test>

has been changed to

    <test-case name="class-range">
      <result>
	<assert-not-a-grammar/>
      </result>
    </test-case>

Unless I am mistaken, this makes the catalog invalid against the schema
in ixml-tests/lib/test-catalog.rnc.

In the schema, grammar-test is distinguished from test-case for a couple
of reasons:

  - A grammar test specifies only one input, not two; the processor
    should parse the specified grammar as an input string against the
    processor's built-in copy of the ixml specification grammar and
    also test the grammar for conformance to other requirements (like
    uniqueness of left-hand sides, or existence of Unicode classes
    referred to in the grammar).

  - From some angles it seems plausible that the set of possible results
    may differ for a grammar test.  (I've gone back and forth on this;
    at the moment I am inclined to think that the main difference is in
    the meaning of the assertions.  As the comments in the schema say,
    in a grammar-test there may not be a useful distinction between
    assert-not-a-grammar and assert-not-a-sentence, whereas in a
    test-case they mean very different things.)
  
  - If a grammar fails a grammar-test, all of the test cases in the
    test-set should normally fail with assert-not-a-grammar.

> I think it would be useful to
> have a repository of discrete tests about which we can report
> conformance and raise issues.

Agreed.

Michael


-- 
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com

Received on Saturday, 5 February 2022 16:21:35 UTC