Re: DoV and verifiability

On Fri, 6 Sep 2002, Lofton Henderson wrote:

> At 10:24 PM 9/5/02 -0600, Alex Rousskov wrote:
>
> >Yes, in general, all variations, including variations in degrees of
> >conformance, are evil in a sense that if a spec can cover the same
> >technology without using variation X, it SHOULD be written without X.
>
> On this we agree.
>
> >I doubt there is need to argue about this "minimal set" requirement.
>
> On this, the QAWG is uncertain.

Your own explanations contradict that there is an uncertainty. You
(and QAWG) seem to be certain that the "minimal set" requirement is a
good valid SHOULD. The uncertainty seems to be related to whether we
should go beyond that simple and generic requirement:

> The "minimal set" issue is:  can we come up with something more
> quantifiable, measurable, or verifiable than "if a spec can cover
> the same technology without using variation X, it SHOULD be written
> without X".  (Which is basically what SpecGL says now, for each of
> the dimensions in DoV).

Right, so it is an optimization problem: a simple and generic
requirement versus a complex set of more specific requirements.

IMO, you cannot come up with something significantly more
"quantifiable, measurable, or verifiable" than a simple SHOULD. You
are just increasing the level of complexity without covering any new
ground. In fact, you are losing ground because you do not (and cannot)
predict all DoV that will exist!

> >I also doubt anybody would seriously object that documenting and
> >analyzing common degrees of variability in existing specs is a good,
> >useful project.
>
> Okay, but from your later comments, I assume that you mean this should not
> be in SpecGL?  I.e., it should be in Specification Example & Techniques, or
> elsewhere.

Correct.

> Also, do you distinguish between establishing (in SpecGL) the
> parameters (dimensions) as organizing points of such analyses, on
> the one hand, and the results of such analyses (somewhere) on the
> other?

I think both do not belong to SpecGL. There is no need to establish a
parameter that the spec itself is not using. This would violate,
again, the "minimum set" requirement.

> I agree.  In particular, we anticipate the following structure when
> we have gotten SpecGL in better editorial shape:  each checkpoint is
> phrased as an imperative statement; each checkpoint contains one or
> more individual requirements; satisfying the checkpoint requires
> satisfying all of the individual requirements.
>
> The requirements can be MUST, SHOULD, or MAY (or their negations).  So for
> example:
>
> Checkpoint X.Y ...blah...
>
> * (requirement)...MUST...
> * (requirement)...SHOULD...
> * (requirement)...SHOULD...

Yes. To follow our own guidelines, I would argue that the requirements
should be numbered/IDed. Checkpoint itself is just a subsection
structure of no normative importance. You do not satisfy/test
Checkpoints, you satisfy/test requirements. There is no need to
introduce a second normative level, IMO. Keep it simple and flat.

> If there is something that (we think) is worth saying, and if we
> find that it is "virtually impossible to objectively prove", then
> why not put it as a SHOULD requirement?  Not matter what a spec
> writer does, that requirement will not lead to failure of the
> checkpoint as a whole.
>
> If that is true, what is the value?  There is a school of thought that a
> lot of the value of SpecGL will be in simply helping editors (and WGs) to
> focus on the key issues and make explicit, defensible choices.  I agree
> with it, having seen (for each of the various of the DoV)  examples in WGs
> where the outcome was somewhat default, and probably not carefully
> considered.  (In fact, I confess -- I'm guilty of some of that.)

Agreed, except I think _we_ should not look at SpecGL as a helping
guide. That is, SpecGL should apply to a given spec, not to the
process of writing specs. Other spec authors will use it as they
please, and yes, it will be helpful as a guide (as a side-effect).

> >         - Is it appropriate for SpecGL to illustrate a "minimal
> >           set" SHOULD with a lengthy discussion about some common
> >           degrees of variability?
>
> You're losing me here.  Can you point to an example in SpecGL?

All sections that talk about DoV are, essentially, an
examples/illustration of the "minimal set" SHOULD. You said it
yourself above: QAWG wants to split a generic SHOULD into a number of
more specific SHOULDs using DoV framework.

> >IMO, no it is not appropriate;
> >           that discussion belongs elsewhere because having it
> >           would violate the "minimal set" SHOULD.
>
> Again, I'm unsure what you're saying.  If it is violating a "minimal
> set"  SHOULD, then it must have an unnecessary dimension from
> amongst the DoV.

The presence of DoV model _itself_ violates the "minimal set"
SHOULD.  The DoV model/discussion is not needed to define a good spec.
Defining a good spec should be the primary (if not the only) purpose
of SpecGL, IMO.

> >The SpecGL spec is not the right place to "try to highlight at least
> >some of the key variables that should be considered by
> >specifications", IMO. SpecGL should contain specific, mostly testable,
> >clauses that can be applied to a given spec. SpecGL should _not_
> >contain instructions how to write a [good] spec. This is one of the
> >key disagreements I have with the current wording/intent.
>
> This here is a key issue and apparent point of disagreement -- the
> question of highlighting the variables to be considered.
>
> Behind the current SpecGL approach is the belief that it is valuable
> and useful for SpecGL to address and bring attention to some things
> (especially amongst the enumerated DoV) that are not testable via
> MUST assertions.  (Or at least we haven't yet found a good MUST
> assertion for them).  This position is based on practical experience
> with W3C (and ISO and ...) specs having defects in one or more of a
> handful of categories -- modularization, examples, discretion,
> extensions, etc... i.e., the DoV.

I agree that it is valuable and useful, but I disagree that SpecGL is
the right place. If it is, it should not be a spec. It should be a
tutorial/article/whatever. We have to make a choice:
	- SpecGL is a spec with MUSTs/SHOULDs/etc.
	- SpecGL is a tutorial

I argue that mixing the two together is a very bad idea because
	- MUSTs/SHOULDs get lost/misinterpreted/ignored in volumes of
	  non-normative text
	- Non-normative text becomes too structured and formal to
	  be pleasant to read/follow

Divide and conquerer.

With DoV framework you are over-engineering/over-optimizing SpecGL.

> Just to make sure I'm understanding you, can you point me to an
> example in SpecGL text of what you mean by "instructions how to
> write a [good] spec"?

Everything in the form "the authors should consider or be aware of
this or that" is an instruction to the authors. The entire DoV model
is only useful if you are writing a spec and looking for instructions
on how to do it and what to avoid. IMO, if SpecGL is a spec, it should
be about other specs (the subject is a spec), not about ways of
writing other specs (the subject is the author of a spec or the
writing process).

> Do you mean by SpecGL, or by specs?  I.e., are you saying that SpecGL
> itself should not address one or more of the dimensions in DoV (in which
> case, which ones)?  Or are you saying that SpecGL should not require specs
> to address them?

SpecGL should not mention DoV model in any normative way. The "minimal
set"  SHOULD is enough. SpecGL introduction should point to DoV
tutorial/paper.

> >My argument is that the
> >"minimal set" SHOULD requirement above is sufficient; no need to
> >invent an "organizing concept".
>
> Actually, we *did* need to invent a refinement and factorization of some
> big fuzzy concepts like "flavors of conformance" from the 20020515 SpecGL
> draft.

The "minimal set" SHOULD requirement already covers "flavors of
conformance". There was no need to invent a refinement and
factorization for that. "Flavors of conformance" is just another
example (SpecGL might mention it as a possible X and that's all).

> That is what I meant by organizing concept:  current SpecGL breaks
> down the wooly "flavors", "levels of conformance", etc into discrete
> and manageable components (parameters, dimensions, etc).
>
> These are the "X" of your "...variation X" above.  So maybe organizing
> concept is the wrong phrase, but I don't think that you're suggesting to
> eliminate the identification of the "X", are you?

I am. X is a meta-level abstraction. You cannot identify it
precisely/completely anyway. You can only give [numerous] examples!
You do not have to call it X, of course. There is probably a better
name for this abstraction.

> As I guess I have made clear, IMO the DoV is a useful approach for
> SpecGL.  I agree that your "'minimal set' SHOULD" ought to be part
> of SpecGL.  I go further and think that other SHOULDs are useful and
> helpful to spec editors, even if they are not objectively verifiable
> (and won't lead to failure of a checkpoint, because they are
> SHOULD).  And I'm sure that we can and should tighten the prose, and
> probably even trim and tighten the particular collection of
> guidelines and checkpoints.

I understand your position. IMO, "useful and helpful [instructions] to
spec editors" should be outside of SpecGL scope. I suggest that, given
a spec S and SpecGL, it would be possible to assert that, with a good
probability,
	- S is AAA conformant (all SpecGL MUSTs and SHOULDs are satisfied)
	- S is AA conformant (all SpecGL MUSTs are satisfied)
	- S is not conformant (at least one SpecGL MUST is violated)
AND that
	- SpecGL is AAA self-conformant

Such a SpecGL would be compact and very useful, especially if there
are a bunch of free-style tutorials to complement it. Tutorials should
talk about how to accomplish AAA conformance (which would, in some
cases, involve handing many DoV issues).

Thanks,


Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Friday, 6 September 2002 13:19:57 UTC