RE: [www-qa] Re: Conformance and Implementations

At 11:24 PM 10/19/2001 -0400, David_Marston@lotus.com wrote:

>Kirill Gavrylyuk writes:
> >David,
> >but would you agree that, while third party may do much better job in
> >creating tests, final decision on spec interpretation should belong to
> >W3C WG, regardless of whether it is the "best interpretation" or not?
>
>Yes, I think you're describing the errata process. It appears that
>Mark Skall is raising a caution that an erratum that reverses the
>meaning of a spec is a Bad Thing. If I am interpreting his reply
>correctly, the WG can know "what it meant" (design rationale) and can
>use the errata process to fill in places where their rationale was
>under-expressed, but if the spec said something contrary to what the
>WG meant, then they (and we) are stuck with what it says.

I'm not sure if I interpret it (Mark's point) that way.  If that's how he 
meant it, then I disagree.  If a mistake in the document has reversed an 
intended consensus meaning, I don't see why we should be "stuck with 
[it]".  There is no reason that an erratum cannot be processed, approved by 
consensus, and thereby establish the intended meaning.

Meanwhile, a test suite should only test what the document 
says.  Especially if it the statement is relatively unambiguous.  As I 
looked at the DOM example that Curt Arnold quoted -- "The system identifier 
of the external subset" -- I thought that actually provided a good example, 
but I reached a different conclusion.  Unless there is some other place in 
DOM where there is an overarching (normative) principle that restricts the 
returned value (no absolutization), then I agree with Mark:  the spec 
wording apparently allows relative or absolute.  (I don't mean to 
completely dismiss Curt's point about "reasonable interpretation", but the 
fact that one implementor did absolutization indicates to me that it is not 
unreasonable.)

Until the spec is corrected with an erratum, the test suite should not 
attempt to impose an interpretation.  Nothing short of a consensus erratum 
process addresses this problem definitively.

(By the way, this illustrates why test suites, on an individual test case 
basis, need to be tied to erratum level -- the OASIS XSLT/Xpath Conformance 
TC, for example, is implementing this in its framework).

If that principle is agreed, then a *fast*, normative, consensus erratum 
process is needed.


>Lofton Henderson adds:
> >However, the length of W3C document cycle, by which corrections
> >and clarifications actually get published, creates another problem.
> >...This suggests the utility of a "normative errata" mechanism.
> >...Defect corrections go through a formal review and resolution
> >process, which is typically much shorter than
> >document-release cycles.
>
>I was arguing for something like this at the April QA workshop.
>Ideally, a developer would stumble across something that was
>under-specified, bring the issue to the WG, and get a *published*
>ruling so quickly that it would be feasible to simply suspend
>development in that area for a few days while awaiting the ruling.

This is exactly right, I think.  Fast can be as quick as:  the WG considers 
the issue and decides a recommended solution;  WG publishes proposed 
erratum; and, have something like a 30 day "PR-like" review, comment, and 
resolution.

In my past experience with ISO Defects process, I was both a committee 
member and a technology implementor.  The process is open and quick enough, 
and the correct resolution is usually obvious early on, so that 
implementation work (software, test suites, etc) could proceed towards the 
anticipated correction.  At worst, as you say, briefly suspend work in the 
given area.

By the way, the attached HTML file is an example of an ISO defect report, 
instantiated in its rigid pro-forma.

>I stress "published" above because all other developers and test
>writers need to be aware of the ruling on a timely basis.

Yes.  This is critical, and it is missing from less formal or ad-hoc 
approaches.  I would include technology implementors, along with "test 
writers".  And the solution needs to quicky be available to *all* such 
people, not only those who are active in the inner circle of the standard's 
development.

>Please
>keep this ideal in mind when designing a fast-turnaround erratum
>process.

I was surprised to discover that there is no such thing.  What the Process 
Document says about "errata" is pretty thin, and implies a complete 
document republication.  Is this (errata) addressed anywhere else in W3C 
(process) documents?


>Beyond that, we need an appeals process. I propose that appeals be
>heard on the grounds of under-specification. You couldn't appeal an
>assertion in the spec on the grounds that it's a bad design, but
>you could appeal if your original query was in a gray area and the
>fast-turnaround response said that sufficient assertions already
>exist. The appealer would present examples where a necessary choice
>must be made (i.e., you can't "do both") and try to show that there
>are not enough assertions. The WG could defend by pointing out more
>assertions that pertain.

Another interesting issue.  I haven't seen much precedent here (except 
testing appeals to a Control Board, in a certification testing environment).

-Lofton

*******************
Lofton Henderson
1919 Fourteenth St., #604
Boulder, CO   80302

Phone:  303-449-8728
Email:  lofton@rockynet.com
*******************