Spec Guideline: subjectivity (re: issue #87, #94, etc.)

Issue #87 on the issues list at
http://www.w3.org/QA/WG/qawg-issues-html
asks whether the dimensions of variablity (DoV) should be refined,
collapsed, or eliminated in the Spec Guidelines (SpecGL). This is one
of the main areas where the QAWG hoped to get feedback on the SpecGL
Working Draft. Issue 94 specifically questioned my proposal that all
specs should be explicit about which DoV they don't use. (I stated at
http://lists.w3.org/Archives/Public/www-qa/2002Sep/0019.html
that I expected the 8 DoV to be a complete set, at least for the
current state-of-the-art in specs that the W3C might recommend.)
Issues 90 and 92 question two specific DoV.

These issues originated from Alex Rousskov's message at
http://lists.w3.org/Archives/Public/www-qa/2002Aug/0060.html
In some sense, he is arguing that the SpecGL should say "write good
specs" and only deal with DoV in the most tangential way. But the
DoV were organized in response to the "variations are bad" thread.
http://lists.w3.org/Archives/Public/www-qa/2002May/0035.html
introduced that thread, which is now Issue #69. So we need a
decision there:
A. SpecGL merely requires that it be possible to distinguish
   implementations that are conforming and non-conforming at a
   structural level. That is, whatever DoV techniques are used
   by an implementation can either be overlooked or reduced to
   something testable.
B. SpecGL enumerates the DoV but does not express any value
   judgement about excessive variability.
C. SpecGL enumerates the DoV and discourages variability beyond
   a certain objective criterion.

I think that we currently suffer at two levels, and SpecGL can help us
in at least one of those levels. Those of us trying to write vendor-
neutral conformance tests (my full-time job) suffer when the spec
allows so much variability that there is little or no common ground
across all conformant implementations. But we also suffer if we try to
create an adaptable test suite, or to write vendor-specific tests that
match the vendor's choices, because it is hard (and potentially
subjective) to derive all the variability permitted by a typical W3C
Rec of today. For example, if an implementer is allowed to implement
one or more of four independent (parallel) modules, we can have an
adaptable test suite that tests each module independently only if all
parties agree on what the modules are. Thus, the identification of the
modules is just as much a requirement on the specs as any other
substantive requirements on behavior. Explicitly defining the modules
as (magic word) "modules" sets conformance expectations at a level
that enhances the ability of independent testing labs to conduct their
own test and achieve consistent results. The specs can and should embody
techniques that will ward off "we conform"/"no you don't" arguments by
vendors and frustrated users. Please note that this benefit (someone
other than the vendor can test whether the implementation conforms)
leads to the benefit of interoperability, but it's separate. Making
choice C above addresses interoperability.

I think we all agree that the SpecGL should define a complete boundary
between conforming and non-conforming implementations. I think a spec
should be able to express requirements of the form "If you choose to
implement X, you must implement it this way." This means that there is
such a thing as constrained discretion, but it also means there is
variability among implementations. QAWG will facilitate the creation of
vendor-neutral test suites and will have to deal with constrained
discretion. To maximize the benefits of open systems, the test suites
and specs should facilitate testing by neutral third parties. Hence
there is a need for clear delineation of variability techniques in use:
modules implemented or not, which choice on discretionary choices,
deprecated items accepted (under their old meaning) or causing an error,
and so forth. It is right and proper for the SpecGL to impose formalisms
on the WG writing the spec, so that multiple implementations can be
tested by an independent lab despite the vendors taking advantage of
allowable variability.

When you get down to details, the spec that employs a particular DoV
must be explicit in doing so. Having a limited set of 8 DoV (with
modules being so generic as to allow widespread variation) helps those
who want to write the testing framework. It would be possible to say
"any variability is a module" but some of the DoV (product classes,
discretionary items, extensions) would be a difficult semantic
adaptation. Does anyone who has been silent up until now want to come
forward with their opinion on how many dimensions there should be?

I suppose that any given test case in a suite can be characterized as
applicable to implementations only if certain variability criteria are
met. Example: this test only applies to Xblah consumers that implement
the audio module, at Level 2, accept deprecated content from Version
3.0 and higher, chose to ignore (rather than error on) negative values
of the foo parameter, and have splatness set on. This test can be a
valid conformance test if a particular outcome is required under those
conditions. But if the spec doesn't "package" the variability using the
DoV techniques or something like them, the test suite would be very
chaotic.

SpecGL can save us from this chaos. Notice however, the necessity that
the test writers know whether *or not* each DoV might be used. Saying
explicitly "there are no modules" is better than saying nothing about
modules and leaving the reader to guess. The conformance clause (or
whatever it will be called) must lay out the entire picture on DoV,
leaving no ambiguity that would lead to implementations that vary in
ways not anticipated by the WG but germane to the functionality the
spec attempts to define. This is issue #94. I think that DoV help the
WG to write an airtight spec, and it's reasonable to have them document
the airtightness by stating the results of their consideration of each
DoV, whether they allowed variation or not.
.................David Marston

Received on Monday, 7 October 2002 15:54:01 UTC