Testing the Penumbra considered beneficial (Issue #23)

From the Issues List [1], issue #23 reads:
"Description: Should tests of MAY/SHOULD assertions be part of a
conformance suite? They are useful, so we need to discuss these tests in
the Framework documents, probably in Test Materials Guidelines and/or
Spec Guidelines. Note that this issue applies to test suites related to
W3C recommendations, not to the Framework documents themselves. Issue
#39 deals with normative language in the Framework documents themselves.
The issue was discussed at the 2002-03-01 March QAWG face-to-face
meeting [2], although the documentation of the discussion seems to refer
more to issue #39. The sense of the discussion was "yes", test for
SHOULD and MAY assertions in technical standards (Recommendations). The
form of the tests and how they enter into the "conform/non-conform"
verdict will depend on how the specification is written, specifically
how it defines conformance related to SHOULD and MAY assertions."

During the discussion later in the week, I offered to organize the
thoughts that came out during the coffee break. Here we go.

Testing of MUST assertions can rightly be deemed "conformance testing"
and falls in the umbra of QAWG's work. Furthermore, a test can be
deemed to be a conformance test if there is some set of circumstances
where non-conforming behavior can be identified. For example, when the
spec offers a two-way discretionary choice, any behavior other than
those two is clearly non-conformant, and even one of the two might be,
depending on how the choice is presented. Contingent conformance, in a
case where specified behavior is required if an optional module is
implemented, represents another scenario in the umbra.

But why does the W3C care about conformance? Not for an end in itself,
but to serve the goal of interoperability ("interop"), which is the
actual benefit to humankind. Interop allows us to choose among various
implementations and the platforms on which they run, and choice opens
the potential benefits of competition.

The W3C has achieved a high level of technical influence without being
a treaty-based standards organization. However, there are some areas
where the W3C does not take a hard stance on interop, though they might
wish to, perhaps due to sensing the limits of their influence. A good
example is W3C's encouragement of UTF-8 and UTF-16 character encodings,
which is typically a SHOULD assertion in W3C Recs. Implementation of
the SHOULD provisions enhances interop. This reveals a penumbra for
product testing: testing for interop. (Testing for MAY assertions also
falls into this penumbra, though reporting results doesn't fit into a
simple pass/fail structure.)

Since many of the test efforts anticipate accepting contributed tests
into a pool, all useful contributions should be accepted and properly
classified. The contributor may think they are submitting a conformance
test, which may get classified as an interop test instead, but it is
still useful and the contributor should be thanked for it. If and when
QAWG designs a test harness, test case management system, or test case
description language, provisions should be made to retain the penumbral
interop tests and run them when interop results are desired.

Proposed Resolution: Tests of MAY/SHOULD assertions should be part of
the *test* suite, but segregated from the true conformance tests. A
test case is deemed to be a conformance test if it can be applied to
multiple implementations, obtain comparable results across the various
implementations, and if a particular set of circumstances exists where
certain results indicate failure to conform. Other tests, herein called
"interoperability tests", can be applied to multiple implementations
and obtain comparable results across the various implementations, but
never indicate conformance failure. The system of running tests should
support both conformance testing (by excluding interop tests) and
interoperability testing.

(Side note: An interop *report* derives data from interop testing
*and* from comparing the Implementation Conformance Statements along
some DoV: modules, profiles, deprecation, discretionaries, and (rarely)
extensions when "published extensions" (SpecGL 9.5) exist. One could
argue that levels belong on the list, but I think that interop reports
would be isolated by level. Obviously, interop reports must always be
isolated by class of product.)
.................David Marston

[1] http://www.w3.org/QA/WG/qawg-issues-html.html
[2] http://www.w3.org/QA/2002/03/01-f2f-minutes

Received on Wednesday, 19 March 2003 13:54:41 UTC