Re: integrating error codes into the test suite and its schema

"C. M. Sperberg-McQueen" <cmsmcq@blackmesatech.com> writes:
> Norm Tovey-Walsh writes:
>
[…]
>> I can’t help but feel like you’re making the problem harder by having an
>> implementation that returns more than one error code. :-)
>>
>> Does the following work/is it equivalent:
>>
>> If a test is expected to fail and no error code is provided, any failure
>> code is a pass.
>
> I am not sure I understand.  Let me try a paraphrase, and you can
> correct what I get wrong.

I tried to respond in detail to your paraphrase, but I got confused by

      For any test with an expected result of assert-not-a-sentence (in an
      instance test), assert-xml, or assert-xml-ref, without an error

because assert-not-a-sentence is an error case in my mind and assert-xml
is not.

Let me try to back up and describe what I have in mind at higher level.
Then you can tell me why I’m wrong :-)

For some tests, call them “should-work” tests, the implementation is
expected to take a grammar and a sentence and produce some XML. For
these tests:

1. An implementation that signals an error, any error, fails the test.
2. An implementation that fails to produce (one of the) expected XML
   results fails the test.
3. An implementation that does not signal an error and produces
   (one of the) expected XML results passes the test.

For some tests, call them “should-not-work” tests, the implementation is
expected to take a grammar (and maybe) a sentence and it is expected to
signal an error. Either the grammar contains an error or the sentence
contains an error. Or the grammar and the sentence taken together should
raise an error.

1. An implementation that produces output without signaling an error,
   fails the test.
2. An implementation that signals an error passes if:
   a. The test does not enumerate any error codes, or
   b. The implementation returns one of the enumerated error codes.

(For 2b, if an implementation produces more than one error code, there’s
a question I guess about whether all the codes produced must be in the
enumerated set or only one of them. I’m not sure what the right answer
is. You choose.)

> If I have understood correctly, I think this worries me; it means that
> for not-a-grammar or dynamic-error cases in which no defined error code
> applies, a processor passes the test even if it supplies an inapplicable
> error code like S11.

The solution to that, I think, is to make sure we enumerate error codes
for all the should-not-work tests. I’d be ok with saying that the error
attribute is *required* on should-not-work tests, it just means the test
suite can’t be valid until we’ve added all those attributes. The only
reason to allow a should-not-work test without any enumerated error
codes, I think, is laziness on our part.

> I should not explicitly that I assume that there are and will be test
> cases for which no error codes are prescribed.  The class of these that
> come to mind off-hand are grammar-tests on inputs with syntax errors.

Right. How accurately and usefully an implementation describes the error
when an input grammar doesn’t parse is going to be…interesting. I’m
inclined to to fret too much if an implementation reports D02 for that
case. I trust users will put pressure on the implementor to do better.

> We could change this situation by supplying error codes for 'It is an
> error if an input grammar is not described by the specification grammar'
> and 'It is an error if an input grammar in XML is not schema-valid
> according to the schema for ixml grammars', but I don't want to assume
> we will do so.

Not for version 1.0 anyway.

> In general, I think I can live with either no-attribute-present or
> error="none" as the signal for "this test has no prescribed error
> codes".  The rationale for entertaining an explicit 'none' signal is:
>
>   1 In general, if you can achieve something by saying nothing, it's
>     better if the language allows you to achieve the same thing by
>     saying something explicit.  I appeal to this in discussions of ^ on
>     terminals, so I should take it seriously.
>
>   2 In the near-term future, we are going to have a lot of tests we have
>     not examined for error codes.  It will be convenient, during the
>     transition, if not having an error attribute means clearly "not yet
>     handled".

I take your point on allowing a grammar to be explicit about things. I
withdraw any objection I raised to a value of “none” for the error
attribute.

>>> There are no extra points for using all the error codes specified, there
>>> are no penalties for using just one.  There *are* penalties for using an
>>> unexpected ixml error code.
>>
>> Exactly so.
>
> That's why I think, if no error codes are mentioned on the test case,
> reporting an ixml error code should probably be a fail or a
> wrong-error.

Ok.

> If we want an 'any error code is allowed' signal (meaning any ixml error
> code -- nothing we do affects what a conforming processor can do with
> its own error codes), I would want to call it 'any' not 'none'.  But do
> we want such a beast?

I wasn’t trying to propose that last time, but don’t we actually need
one for the cases mentioned above where the spec doesn’t provide an
error code?

> Or are you just being acquiescent because you thought I wanted one?

You’re doing the heavy lifting here, I’m happy to just follow your lead
on most things.

> Actually, you wrote the memo [1].  
>
> [1] https://lists.w3.org/Archives/Public/public-ixml/2022Apr/0029.html

That explains it then. I don’t read the drivel I write. :-)

                                        Be seeing you,
                                          norm

--
Norm Tovey-Walsh
Saxonica