Re: integrating error codes into the test suite and its schema

> I will try to respond in more detail later, but for now I'll just say we
> do seem to have a terminological issue.

Right. Okay. Let’s see if I can enumerate the possibilities.

For a given grammar G which defines a language L(G) and an input S:

0. The user might fail to provide usable inputs. The files might not
   be found, the URIs might 404, the input might have encoding errors.
   Let’s say those problems are out-of-scope.
  
1. G might be syntactically invalid.

2. G might be syntactically valid but violate some semantic
   constraint (e.g., that a nonterminal must have at most one
   definition).

3. G might be syntactically and semantically valid.

The input S is only relevant in case 3, I think.

4. S might not be in L(G)

5. Analysis of S might cause the processor to fail (due to insufficient
   memory, for example)

6. S might be in L(G) but produce an invalid serialization.

7. S might be in L(G) and produce a well-formed XML serialization.

Does that cover all the cases?

> I think "error" is not the correct term for all cases in the second
> class.  If we have to have one, 'failure' might work.  I sometimes refer
> to 'abnormal termination', which is probably confusing to some or many
> people.

(Everyone who worked on IBM mainframes at one time, put the return code
in R15 and say “ABEND”.)

My framing was between tests that are expected to pass (case 7) and
tests that are expected to fail (cases 1, 2, 4, and 6). I guess that
boils down to saying “pass” means produces XML output and does not
signal an error. That doesn’t define a term for the case where
everything is valid but S is not in L(G), but I don’t think the
processor produces any XML in that case, so it’s covered.

For case 7, an implementation passes by not signaling any errors and
producing (one of) the expected XML serializations.

For cases 1, 2, 4, and 6, it passes only if it signals an error and
either a or b:

  a. The test does not enumerate the acceptable errors. (Or
     alternatively, explicitly says that any error is acceptable.)

  b. The test does enumerate acceptable errors and the
     implementation produced one of them.

For cases 1, 2, 4, and 6, if the processor does not signal an error, it
fails the test. If it fails to produce an acceptable error, it
wrong-errors the test.

I don’t think there are any tests that only test for 3, but I don’t
think there need to be.

I don’t think there are any tests that test for 5, but I don’t think we
can write any such tests in an implementation independent manner. I’ve
never caused an out of memory error. I have some grammars that might,
but I’ve never let them run to completion because I’m not sure there’s
enough time left before the heat death of the Universe.

                                        Be seeing you,
                                          norm

--
Norm Tovey-Walsh
Saxonica