Re: integrating error codes into the test suite and its schema

Norm Tovey-Walsh writes:

>> Suggestion 1a: with an attribute on the assertion(s).
>
> That seems fine to me, though I don’t immediately see the value of
> putting the error attribute in the ixml namespace. Surely, in the ixml
> test suite, the meaning of an unqualified errors attribute is that it’s
> the ixml errors. No?

Works for me.  Let's call the attribute {}error.

>> Suggestion 2a: In the expected result, specify all applicable error
>> codes, with the meaning "one or more of these is expected".

> This one.

OK.

>> Suggestion 2b: If suggestion 2a is adopted, the order of codes given
>> carries no significance; they are a set.
>
> Agreed.

Good so far.

>> Q3 How do the codes affect reported results?
> […]
>> When we were discussing error codes, Norm argued (if I understood him
>> correctly) that in the XQuery/XSLT case getting the error codes wrong
>> was more often a symptom of a bug in a processor than a symptom of a
>> different analysis of what was wrong with the input. 
>
> I was describing XProc error codes, but yes.
>
>> So I don't want to treat wrong-code results as passes.
>
> I can’t help but feel like you’re making the problem harder by having an
> implementation that returns more than one error code. :-)
>
> Does the following work/is it equivalent:
>
> If a test is expected to fail and no error code is provided, any failure
> code is a pass.

I am not sure I understand.  Let me try a paraphrase, and you can
correct what I get wrong.

    For any test with an expected result of <assert-not-a-grammar/> or
    <assert-dynamic-error/>, and for any grammar-test with a result of
    <assert-not-a-sentence/>, with no error attribute, a processor which
    signals the appropriate abnormal termination passes the test,
    whether they report any ixml error code or not.

And, I guess, implicitly:

    For any test with an expected result of assert-not-a-sentence (in an
    instance test), assert-xml, or assert-xml-ref, without an error
    attribute, a processor which reports an ixml error code fails the
    test.

If we decide that we want an explicit signal that there are no error
codes (so we decide to code those two expected results as
<assert-not-a-grammar error="none"/> and <assert-dynamic-error
error="none"/>), the first rule would apply in those cases.  (More on
the topic of error="none" below.)

If I have understood correctly, I think this worries me; it means that
for not-a-grammar or dynamic-error cases in which no defined error code
applies, a processor passes the test even if it supplies an inapplicable
error code like S11.

I should not explicitly that I assume that there are and will be test
cases for which no error codes are prescribed.  The class of these that
come to mind off-hand are grammar-tests on inputs with syntax errors.

We could change this situation by supplying error codes for 'It is an
error if an input grammar is not described by the specification grammar'
and 'It is an error if an input grammar in XML is not schema-valid
according to the schema for ixml grammars', but I don't want to assume
we will do so.

> If a test is expected to fail and a list of error codes is provided, the
> test passes if all of the error codes reported by the implementation are
> in the list.
>
>    [ On a second reading of this message, I see the point above fails to
>      account for the possibility of the explicit code “none”. But then I
>      think, “what does ‘(ixml:)errors="none D01"’ mean?” and I feel like
>      maybe the explicit value “none” actually introduces more
>      possibilities for error, but I’m not going to fuss over it. ]

If we use 'none', then it makes sense only as the sole token; I believe
RNG should be able to handle that.

In general, I think I can live with either no-attribute-present or
error="none" as the signal for "this test has no prescribed error
codes".  The rationale for entertaining an explicit 'none' signal is:

  1 In general, if you can achieve something by saying nothing, it's
    better if the language allows you to achieve the same thing by
    saying something explicit.  I appeal to this in discussions of ^ on
    terminals, so I should take it seriously.

  2 In the near-term future, we are going to have a lot of tests we have
    not examined for error codes.  It will be convenient, during the
    transition, if not having an error attribute means clearly "not yet
    handled".

> If the test fails and a list is provided and the implementation
> generates any error codes that are not in the list, the result-type is
> “wrong-error”.
>
> I think wrong-error is better than half-pass.

I can live with that.  And it's more informative.

>> Note in particular that if we have
>>
>>     <assert-dynamic-error ixml:error="D01 D02"/>
>>
>> a processor passes by reporting either error D01 or error D02 or both,
>> and no other ixml errors.  (It may report implementation-defined
>> errors.)
>
> Are implementation-defined errors overkill here? If the implementation
> wants to report D01 and impl:a-specific-reason-for-D01, can it not have
> a separate channel for the more specific reasons? Do they have to be
> accounted for in the test suite?

I do want the test suite to be able to handle them, but as you point
out, impl1:error="a b c" impl2:error="QXYZ0042 QXYZ0043" works just fine
for that.


>> There are no extra points for using all the error codes specified, there
>> are no penalties for using just one.  There *are* penalties for using an
>> unexpected ixml error code.
>
> Exactly so.

That's why I think, if no error codes are mentioned on the test case,
reporting an ixml error code should probably be a fail or a wrong-error.

>> If I can get working consensus on this, I will update the schema for
>> test catalogs in ixml/schemas to require ixml:error attributes on
>> appropriate elements.  The value 'none' will mean that there no relevant
>> ixml codes: e.g. for bad grammar syntax.  Unless someone objects, I will
>> define 'working consensus' as one affirmative assent and no objections,
>> and objectors will have 3 days from today to object, unless there is no
>> assent during that period, in which case the first reaction will win.
>
> If you feel strongly about putting the error attribute in the ixml
> namespace, I won’t object, but it seems unnecessary to me.

OK, I'm sold.  No namespace.

> Likewise, I don’t object to using “none” to mark a test explicitly with
> the semantics “any error code is allowed”, but equally, I’d be happy to
> say that that is the semantic of not providing an (ixml:)errors
> attribute.

If we want an 'any error code is allowed' signal (meaning any ixml error
code -- nothing we do affects what a conforming processor can do with
its own error codes), I would want to call it 'any' not 'none'.  But do
we want such a beast?  Or are you just being acquiescent because you
thought I wanted one?


>> After that, I propose that we systematically update the test catalogs to
>> provide ixml:error attribute on all assertions involving abnormal
>> termination, and that at the same time we update the test catalogs to
>> use the new namespace.  So old namespace = not-updated, new namespace =
>> updated.
>
> Did I miss a memo? This is the first mention I see of changing the test
> catalog namespace. But what you suggest seems like a fine plan to me in
> any event.

Actually, you wrote the memo [1].  

[1] https://lists.w3.org/Archives/Public/public-ixml/2022Apr/0029.html

This will be the first change to the test-catalog schema since I copied
it over from cmscmq/ixml-tests, so it will be the first one to require
the catalog documents (and our test harnesses) to shift to the new
namespace.

Michael

-- 
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com

Received on Wednesday, 25 May 2022 17:24:07 UTC