Re: Conformance Disclaimer

>[...]
>http://www.w3.org/QA/WG/qawg-issues-html#x105
>
>Discussion is welcome, on this (IG) mail list.
>[...]
>At 12:24 PM 12/11/02 -0500, Mark Skall wrote:
>>
>>The conformance disclaimer currently states "Failing to achieve 
>>conformance degree of at least A-conforming does not mean that the 
>>subject specification is necessarily deficient to its intended purposes, 
>>nor does it mean that it is an unacceptable basis for the development of 
>>quality test materials."

History/Background.  This goes back a long way.  It is even in the May 15 
first PWD of SpecGL,

[1] http://www.w3.org/TR/2002/WD-qaframe-spec-20020515/#b2ab3d427,

as well as the published OpsGL of the same date.  It is my recollection 
that it was added as a consequence of OpsGL checkpoint 6.4, which in the 
most recent published OpsGL looks like

[2] http://www.w3.org/TR/2002/WD-qaframe-ops-20021108/#Ck-provide-disclaimer.

I think there are some problems with the statements [1] as a fulfillment of 
CP6.4 [2] (see below).  The disclaimers were probably also strongly 
influenced by and written to reflect the sense of similar statements in, 
for example, UAAG 1.0,

[3] http://www.w3.org/TR/2002/REC-UAAG10-20021217/conformance.html#Conformance

To quote from [3]:
"The UAWG expects conformance this document to be a strong indicator of 
accessibility, but it may be neither a necessary nor a sufficient condition 
for ensuring the accessibility of software. Thus, some software may not 
conform to this document but still be accessible to some users with 
disabilities. Conversely, some software may conform to this document but 
still be inaccessible to some users with disabilities."
>>I believe this undermines our attempt to get WGs, recs and test suites to 
>>adhere to, at least Level A.

If it is not carefully written, there is that potential.

However, I tend to think that the statements are true.  At [1], the 
statement #1 says that conformance to TestGL (or SpecGL, or OpsGL) is not a 
sufficient condition to guarantee the suitability of the test materials (or 
...).  And statement #2 says that A-conformance is not a necessary 
condition for test materials to be useful to their intended purpose (or 
...).  In other words, these disclaimers are modelled closely on the 
similar statements in WAI.

Here is the problem that I see with the statements [1] being a fulfillment 
of the CP6.4 [2].  CP6.4 says:
### start quote ###
Checkpoint 6.4. Provide a conformance verification disclaimer with the test 
materials.

To fulfill this checkpoint, the Working Group MUST provide a disclaimer 
about the use of the test materials for conformance verification of 
implementations.

Discussion. Although tests suites may be used for conformance verification, 
the Working Group must make users aware that:

    1. passing all of the tests does not guarantee full conformance of an 
implementation to the specification
    2. failing the test suite means failing tests for the specific feature 
they target"
### end quote ###
Let's take the case of OpsGL, to avoid confusion about which specification 
or which test materials we're talking about.  OpsGL is a specification 
(standard) that defines requirements for WG operations and 
processes.  CP6.4 requires a disclaimer about test materials and 
conformance to the specification to which the test materials correspond.

What are the test materials for OpsGL?  I don't know.  But OpsGL's 
conformance disclaimer makes no reference to test materials and how 
passing/failing the test materials relates to OpsGL conformance.   It talks 
about OpsGL conformance and how that relates to the suitability of the WG 
processes and operations.

>>In fact, it may be in direct contradiction to this goal. Please document 
>>an issue to revisit the wording for the disclaimer.   The options would 
>>be: 1)leave disclaimer as is; 2)reword this part of the disclaimer; 
>>3)eliminate this part of the disclaimer

Notwithstanding that the conformance disclaimers such as [1] have a problem 
in relationship to CP6.4, I tend to think that the "neither necessary nor 
sufficient" statements are true.

Which leads me to believe that either #1 or #2 is the proper solution.  I 
guess I lean toward #2 -- UAAG said it well, IMO.

And, unless someone has a good idea about what are the test materials for 
OpsGL (and SpecGL, and TestGL), then we should be careful to NOT imply that 
the (edited) statements are a satisfaction of CP6.4.  The latter would only 
be true if OpsGL were the test materials for OpsGL.

-Lofton.