Re: integrating error codes into the test suite and its schema

Norm Tovey-Walsh writes:

> "C. M. Sperberg-McQueen" <cmsmcq@blackmesatech.com> writes:
>> Norm Tovey-Walsh writes:
>>
> […]
>>> Does the following work/is it equivalent:
>>>
>>> If a test is expected to fail and no error code is provided, any failure
>>> code is a pass.
>>
>> I am not sure I understand.  Let me try a paraphrase, and you can
>> correct what I get wrong.
>
> I tried to respond in detail to your paraphrase, but I got confused by
>
>       For any test with an expected result of assert-not-a-sentence (in an
>       instance test), assert-xml, or assert-xml-ref, without an error
>
> because assert-not-a-sentence is an error case in my mind and assert-xml
> is not.

For assert-not-a-sentence, we do need to signal an distinct condition
(e.g. with ixml:state and possibly with other information we emit) and
we may wish to provide diagnostics.  For assert-xml and assert-xml-ref,
it is the usual, standard, normal, and/or unmarked condition and it
would be odd to provide anything one might call diagnostics.  ("This
worked normally.  How on earth did *that* happen?")


> Let me try to back up and describe what I have in mind at higher level.
> Then you can tell me why I’m wrong :-)

> For some tests, call them “should-work” tests, the implementation is
> expected to take a grammar and a sentence and produce some XML. For
> these tests:

> 1. An implementation that signals an error, any error, fails the test.
> 2. An implementation that fails to produce (one of the) expected XML
>    results fails the test.
> 3. An implementation that does not signal an error and produces
>    (one of the) expected XML results passes the test.

I take your class of "should-work" tests to be those tests whose
expected result is assert-xml or assert-xml-ref.

And for concreteness I will take 'signaling an error' to mean reporting
one (or more) of the ixml error codes in the spec.

If that is correct, then I think this is correct.  I note that the way I
just defined 'error' here means that a parser that fails on resource
issues or for other extraneous reasons fails the test because of 2, not
because of 1.

I am nervous about saying "yes this makes sense" because I am clearly
making a lot of communication errors in this exchange, and apparent
agreement may merely reflect yet another failure to understand what you
mean.  But modulo that risk, yes, this makes sense.


> For some tests, call them “should-not-work” tests, the implementation is
> expected to take a grammar (and maybe) a sentence and it is expected to
> signal an error. Either the grammar contains an error or the sentence
> contains an error. Or the grammar and the sentence taken together should
> raise an error.

Here we diverge.

If signaling an error is reporting one of the error codes in the spec,
then, again, a processor reporting that 'a' is not a sentence in the
language defined by the grammar "S='s'." will not be signaling an error,
because there is no error code for such a case.

If we define an error more broadly as any failure to conform to the
spec, which I believe is the only sane definition, then failure to
produce a parse tree for the string 'a' against the grammar "S='s'." is
again not an error.  There *are* some errors (in this sense) for which
the spec provides no error code: if the user supplies the string 'a' as
the input grammar and "S='s'." as the input string, the grammar supplied
does not conform to the spec because it is not syntactically correct.  I
don't see an error code for that case, and I believe the current design
tries to use error codes only for cases where conformance requirements
go beyond syntactic correctness.  (If we want an error code for every
failure to conform to the spec, then we have some gaps to fill.)

Since I don't see error codes for all failures of conformance (syntactic
correctness of the input grammar being the one I have in mind - there
might be others), I do not expect the implementation to return an ixml
error code for every error, although I do expect the conformance failure
to be signaled somehow.

And for failures to parse which do not involve failures of conformance,
I don't expect a parser to signal an 'error' in either sense mentioned
above.  Again, the parser needs to signal the failure to parse the
sentence, but since that is the correct and required result, I find it
bewildering to call it signaling an error.


> 1. An implementation that produces output without signaling an error,
>    fails the test.
> 2. An implementation that signals an error passes if:
>    a. The test does not enumerate any error codes, or
>    b. The implementation returns one of the enumerated error codes.
>
> (For 2b, if an implementation produces more than one error code, there’s
> a question I guess about whether all the codes produced must be in the
> enumerated set or only one of them. I’m not sure what the right answer
> is. You choose.)

I'm not completely sure what the right answer is, either.  The proposal
in my initial note was that all the ixml error codes reported by the
processor must be in the set of expected ixml error codes, and I think
I'll stand by that.  (And note that I mean that to include the case
where the test catalog lists no error codes: if no ixml error codes are
listed, a processor that reports an ixml error code does not pass the
test.)

>
>> If I have understood correctly, I think this worries me; it means that
>> for not-a-grammar or dynamic-error cases in which no defined error code
>> applies, a processor passes the test even if it supplies an inapplicable
>> error code like S11.

> The solution to that, I think, is to make sure we enumerate error codes
> for all the should-not-work tests. I’d be ok with saying that the error
> attribute is *required* on should-not-work tests, it just means the test
> suite can’t be valid until we’ve added all those attributes. The only
> reason to allow a should-not-work test without any enumerated error
> codes, I think, is laziness on our part.

OK.  I think that means I misunderstood the intent of the current set of
error codes.  We are missing

  S99 It is an error if the grammar is given in ixml and is not
  described by the ixml grammar for grammars in this specification.

  S98 It is an error if the grammar is given in XML and is not
  schema-valid against the schema provided in this specification.

But these still do not cover the case of assert-not-a-sentence in an
instance test (a test-case element).  The input string can be determined
to be a non-sentence only with respect to some conforming grammar.  So
by hypothesis the grammar is conformant.  The spec does not define
conformance for input strings, and the parser is behaving correctly in
reporting that the input string is not a sentence in the language
defined by the input grammar.  So ... where is the error?  Where is the
failure to conform to the rules?

>> I should not explicitly that I assume that there are and will be test
>> cases for which no error codes are prescribed.  The class of these that
>> come to mind off-hand are grammar-tests on inputs with syntax errors.

> Right. How accurately and usefully an implementation describes the error
> when an input grammar doesn’t parse is going to be…interesting. I’m
> inclined to to fret too much if an implementation reports D02 for that
> case. I trust users will put pressure on the implementor to do better.

Wait.

I think we may not be talking about the same thing.  I am talking about
a case like this:

  <tc:test-set name="demo-05-26a">
    <tc:created by="cmsmcq" on="2022-05-26"/>
    <tc:description>
      <tc:p>A grammar which is syntactically incorrect.</tc:p>
    </tc:description>
    <tc:ixml-grammar>{ Hi, mom. }</tc:ixml-grammar>
    <tc:grammar-test>
      <tc:result>
        <tc:assert-not-a-sentence/>
      </
    </
  </

The ixml grammar supplied has no rules and thus is not described by the
spec grammar.

I don't any existing error code applies to this situation; if we want
error codes to cover this situation, then the S99 proposed above.

>> We could change this situation by supplying error codes for 'It is an
>> error if an input grammar is not described by the specification grammar'
>> and 'It is an error if an input grammar in XML is not schema-valid
>> according to the schema for ixml grammars', but I don't want to assume
>> we will do so.
>
> Not for version 1.0 anyway.

If we don't, then we will have cases in which the input grammar does not
conform to the spec for which there is no applicable error code.

>> If we want an 'any error code is allowed' signal (meaning any ixml error
>> code -- nothing we do affects what a conforming processor can do with
>> its own error codes), I would want to call it 'any' not 'none'.  But do
>> we want such a beast?
>
> I wasn’t trying to propose that last time, but don’t we actually need
> one for the cases mentioned above where the spec doesn’t provide an
> error code?

I have come around to the view that the error attribute on the assertion
elements is for ixml error codes and nothing else.  Anything else the
processor or developer thinks or talks of as an 'error' (e.g. because
signaling involved calling a function called 'error()') is out of scope.

And for cases where the spec does not provide an error code, using one
of the spec-provided error codes should produce a wrong-error result.

Thank you for being willing to engage on this.

Michael

-- 
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com

Received on Thursday, 26 May 2022 14:58:38 UTC