W3C home > Mailing lists > Public > www-qa@w3.org > September 2002

DoV and verifiability (was: Re: Should SpecGL be a spec?)

From: Lofton Henderson <lofton@rockynet.com>
Date: Fri, 06 Sep 2002 09:50:38 -0600
Message-Id: <5.1.0.14.2.20020906083033.02b5b510@rockynet.com>
To: Alex Rousskov <rousskov@measurement-factory.com>
Cc: www-qa@w3.org

At 10:24 PM 9/5/02 -0600, Alex Rousskov wrote:
>On Thu, 5 Sep 2002, Lofton Henderson wrote:
>
> > In the 20020515 SpecGL WD [1], we had some vague statements about
> > "flavors of conformance", and it raised some discussion on this list
> > that flavors are evil (my paraphrase), and that SpecGL needs to
> > discourage unnecessary variations and flavors.  Before we could even
> > argue the issue about the latter, we needed to clarify what we meant
> > by "flavors".
>
>"Kinds" or "degrees" of conformance, I guess.

After factoring the issue for some time in meetings and telecons, we 
decided that it (variations) needed to reflect not just kinds or degrees 
(levels, categories, etc), but also the other sorts of variations that we 
include under the collective label "dimensions of variability" (DoV).

It is possible that you are including those in your model of "kinds" or 
"degrees". Our factorization is not necessarily the best or only one, but 
we (QAWG) think it is a useful one (on this, we want more input and comments).


>Yes, in general, all variations, including variations in degrees of
>conformance, are evil in a sense that if a spec can cover the same
>technology without using variation X, it SHOULD be written without X.

On this we agree.

>I doubt there is need to argue about this "minimal set" requirement.

On this, the QAWG is uncertain.  Indeed, putting this question out for 
discussion is one of our principal motives behind this WD (along with the 
fundamental question of whether the DoV is a useful factorization and 
organizing concept for the problem).

There is no disagreement with the SHOULD level of normativitiy.  The 
"minimal set" issue is:  can we come up with something more quantifiable, 
measurable, or verifiable than "if a spec can cover the same technology 
without using variation X, it SHOULD be written without X".  (Which is 
basically what SpecGL says now, for each of the dimensions in DoV).


>I also doubt anybody would seriously object that documenting and
>analyzing common degrees of variability in existing specs is a good,
>useful project.

Okay, but from your later comments, I assume that you mean this should not 
be in SpecGL?  I.e., it should be in Specification Example & Techniques, or 
elsewhere.  Also, do you distinguish between establishing (in SpecGL) the 
parameters (dimensions) as organizing points of such analyses, on the one 
hand, and the results of such analyses (somewhere) on the other?


>There are two questions worth arguing about:
>
>         - If SpecGL contains the above "minimal set" SHOULD, how can
>           we test it given a spec? I would argue that one cannot test
>           this requirement in many cases. That is, it is often
>           virtually impossible to objectively prove that X can be
>           deleted without harm. If we cannot test, should we still
>           have this SHOULD? I think so.

I agree.  In particular, we anticipate the following structure when we have 
gotten SpecGL in better editorial shape:  each checkpoint is phrased as an 
imperative statement; each checkpoint contains one or more individual 
requirements; satisfying the checkpoint requires satisfying all of the 
individual requirements.

The requirements can be MUST, SHOULD, or MAY (or their negations).  So for 
example:

Checkpoint X.Y ...blah...

* (requirement)...MUST...
* (requirement)...SHOULD...
* (requirement)...SHOULD...

If there is something that (we think) is worth saying, and if we find that 
it is "virtually impossible to objectively prove", then why not put it as a 
SHOULD requirement?  Not matter what a spec writer does, that requirement 
will not lead to failure of the checkpoint as a whole.

If that is true, what is the value?  There is a school of thought that a 
lot of the value of SpecGL will be in simply helping editors (and WGs) to 
focus on the key issues and make explicit, defensible choices.  I agree 
with it, having seen (for each of the various of the DoV)  examples in WGs 
where the outcome was somewhat default, and probably not carefully 
considered.  (In fact, I confess -- I'm guilty of some of that.)


>         - Is it appropriate for SpecGL to illustrate a "minimal
>           set" SHOULD with a lengthy discussion about some common
>           degrees of variability?

You're losing me here.  Can you point to an example in SpecGL?

>IMO, no it is not appropriate;
>           that discussion belongs elsewhere because having it
>           would violate the "minimal set" SHOULD.

Again, I'm unsure what you're saying.  If it is violating a "minimal set" 
SHOULD, then it must have an unnecessary dimension from amongst the 
DoV.  Which of the DoV would such a discussion (illustrating minimal set 
SHOULD) fit under?


> > DoV is how we are trying to organize the discussion of flavors --
> > the DoV are the underpinnings of the flavors, if you will.  DoV are
> > how we are trying to highlight at least some of the key variables
> > that should be considered by specifications.  No, we won't hit all
> > possible variables, but I think we have captured a good bit of
> > current W3C practice (as well as ISO and other venues) in the
> > current factorization.
>
>The SpecGL spec is not the right place to "try to highlight at least
>some of the key variables that should be considered by
>specifications", IMO. SpecGL should contain specific, mostly testable,
>clauses that can be applied to a given spec. SpecGL should _not_
>contain instructions how to write a [good] spec. This is one of the
>key disagreements I have with the current wording/intent.

This here is a key issue and apparent point of disagreement -- the question 
of highlighting the variables to be considered.

Behind the current SpecGL approach is the belief that it is valuable and 
useful for SpecGL to address and bring attention to some things (especially 
amongst the enumerated DoV) that are not testable via MUST assertions.  (Or 
at least we haven't yet found a good MUST assertion for them).  This 
position is based on practical experience with W3C (and ISO and ...) specs 
having defects in one or more of a handful of categories -- modularization, 
examples, discretion, extensions, etc... i.e., the DoV.

Just to make sure I'm understanding you, can you point me to an example in 
SpecGL text of what you mean by "instructions how to write a [good] spec"?

> > The usefulness of the DoV as an organizing concept for SpecGL is a key
> > issue that we are putting out for discussion with this working draft.
>
>Yes, as long as this issue does not assume that various degrees must
>be covered/described by SpecGL at all.

Do you mean by SpecGL, or by specs?  I.e., are you saying that SpecGL 
itself should not address one or more of the dimensions in DoV (in which 
case, which ones)?  Or are you saying that SpecGL should not require specs 
to address them?

>My argument is that the
>"minimal set" SHOULD requirement above is sufficient; no need to
>invent an "organizing concept".

Actually, we *did* need to invent a refinement and factorization of some 
big fuzzy concepts like "flavors of conformance" from the 20020515 SpecGL 
draft.  That is what I meant by organizing concept:  current SpecGL breaks 
down the wooly "flavors", "levels of conformance", etc into discrete and 
manageable components (parameters, dimensions, etc).

These are the "X" of your "...variation X" above.  So maybe organizing 
concept is the wrong phrase, but I don't think that you're suggesting to 
eliminate the identification of the "X", are you?

As I guess I have made clear, IMO the DoV is a useful approach for 
SpecGL.  I agree that your "'minimal set' SHOULD" ought to be part of 
SpecGL.  I go further and think that other SHOULDs are useful and helpful 
to spec editors, even if they are not objectively verifiable (and won't 
lead to failure of a checkpoint, because they are SHOULD).  And I'm sure 
that we can and should tighten the prose, and probably even trim and 
tighten the particular collection of guidelines and checkpoints.

Regards,
-Lofton.
Received on Friday, 6 September 2002 11:49:57 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:13:59 GMT