Re: Spec Guideline: non-use of a DoV (re: issue #87, #94, etc.)

In the longer message, I wrote:
>> Notice however, the necessity that the test writers know whether *or
>> not* each DoV might be used. Saying explicitly "there are no
>> modules" is better than saying nothing about modules and leaving the
>> reader to guess.

To which Alex Rousskov replied:

>If the spec in question is SpecGL-compliant, then it does not have
>modules if modules are not specifically introduced/defined. And
>vice-versa, if the spec is SpecGL-compliant and uses modules, then
>they must be specifically introduced/defined. This goes from general
>rules/requirements, not specific to DoV.

And how does the reader ascertain that "modules are not specifically
introduced/defined"? If the spec at least satisfies Checkpoint 10.2,
then the reader can just read the conformance section rather than
scan the entire document trying to find all the DoV that are in use.
Still, I contend that an explicit declaration of non-use of each
DoV is better than leaving the reader to determine the situation
by scanning and finding nothing. A good compromise here is to
require the conformance clause to state "all products must be the
same except for the following permissible variations" and then list
all allowable variability.

There was a fine point in my earlier advocacy of particular DoV as
opposed to allowing each WG to name and define their own forms of
variability:
>When you get down to details, the spec that employs a particular
>DoV must be explicit in doing so. Having a limited set of 8 DoV
>(with modules being so generic as to allow widespread variation)
>helps those who want to write the ***testing framework.***

In other words, allowing each WG to re-factor variability for
their spec may not be too damaging to the test suite they produce
(or cause to be produced) in association with their spec, but it
places an enormous burden on those who will produce the testing
framework. A framework can be designed to select test cases by
level, discretionary choices, etc. as long as the designers
know how each dimension will work. If variability is generic
and wide open, who takes responsibility for composing the test
case annotation scheme that allows the test cases to be
filtered by applicability? The WG has to be partly responsible
if they invented new dimensions or mixed/matched known
dimensions in new ways, but they will probably need experts from
QAWG or QA Team to consult on the annotation scheme. Also, all
test cases will need full variability annotation for all the
dimensions, even when a large common core exists, unless the
entire range of variability is airtight and the annotation
scheme is correctly designed.

One troublesome aspect of variability is a subset of the
"discretionary" dimension: wide-open discretion about the
languages and character sets of generated XML. For some specs,
it may be permissible to parse away these differences, while
the correctness of the characters will be a conformance
criterion for others. Does anyone have any suggestions about
controlling the test framework or test cases to obtain
isolation of just the applicable cases?
.................David Marston

Received on Tuesday, 8 October 2002 00:40:25 UTC