Re: integrating error codes into the test suite and its schema

> Suggestion 1a: with an attribute on the assertion(s).

That seems fine to me, though I don’t immediately see the value of
putting the error attribute in the ixml namespace. Surely, in the ixml
test suite, the meaning of an unqualified errors attribute is that it’s
the ixml errors. No?

> Suggestion 2a: In the expected result, specify all applicable error
> codes, with the meaning "one or more of these is expected".

This one.

> Suggestion 2b: If suggestion 2a is adopted, the order of codes given
> carries no significance; they are a set.

Agreed.

> Q3 How do the codes affect reported results?
[…]
> When we were discussing error codes, Norm argued (if I understood him
> correctly) that in the XQuery/XSLT case getting the error codes wrong
> was more often a symptom of a bug in a processor than a symptom of a
> different analysis of what was wrong with the input. 

I was describing XProc error codes, but yes.

> So I don't want to treat wrong-code results as passes.

I can’t help but feel like you’re making the problem harder by having an
implementation that returns more than one error code. :-)

Does the following work/is it equivalent:

If a test is expected to fail and no error code is provided, any failure
code is a pass.

If a test is expected to fail and a list of error codes is provided, the
test passes if all of the error codes reported by the implementation are
in the list.

   [ On a second reading of this message, I see the point above fails to
     account for the possibility of the explicit code “none”. But then I
     think, “what does ‘(ixml:)errors="none D01"’ mean?” and I feel like
     maybe the explicit value “none” actually introduces more
     possibilities for error, but I’m not going to fuss over it. ]

If the test fails and a list is provided and the implementation
generates any error codes that are not in the list, the result-type is
“wrong-error”.

I think wrong-error is better than half-pass.

> Note in particular that if we have
>
>     <assert-dynamic-error ixml:error="D01 D02"/>
>
> a processor passes by reporting either error D01 or error D02 or both,
> and no other ixml errors.  (It may report implementation-defined
> errors.)

Are implementation-defined errors overkill here? If the implementation
wants to report D01 and impl:a-specific-reason-for-D01, can it not have
a separate channel for the more specific reasons? Do they have to be
accounted for in the test suite?

> There are no extra points for using all the error codes specified, there
> are no penalties for using just one.  There *are* penalties for using an
> unexpected ixml error code.

Exactly so.

> If I can get working consensus on this, I will update the schema for
> test catalogs in ixml/schemas to require ixml:error attributes on
> appropriate elements.  The value 'none' will mean that there no relevant
> ixml codes: e.g. for bad grammar syntax.  Unless someone objects, I will
> define 'working consensus' as one affirmative assent and no objections,
> and objectors will have 3 days from today to object, unless there is no
> assent during that period, in which case the first reaction will win.

If you feel strongly about putting the error attribute in the ixml
namespace, I won’t object, but it seems unnecessary to me.

Likewise, I don’t object to using “none” to mark a test explicitly with
the semantics “any error code is allowed”, but equally, I’d be happy to
say that that is the semantic of not providing an (ixml:)errors
attribute.

> After that, I propose that we systematically update the test catalogs to
> provide ixml:error attribute on all assertions involving abnormal
> termination, and that at the same time we update the test catalogs to
> use the new namespace.  So old namespace = not-updated, new namespace =
> updated.

Did I miss a memo? This is the first mention I see of changing the test
catalog namespace. But what you suggest seems like a fine plan to me in
any event.

                                        Be seeing you,
                                          norm

--
Norm Tovey-Walsh
Saxonica