Re: Spec Guideline: subjectivity (re: issue #87, #94, etc.)

On Mon, 7 Oct 2002, David Marston/Cambridge/IBM wrote:

> So we need a decision there:
> A. SpecGL merely requires that it be possible to distinguish
>    implementations that are conforming and non-conforming at a
>    structural level. That is, whatever DoV techniques are used
>    by an implementation can either be overlooked or reduced to
>    something testable.
> B. SpecGL enumerates the DoV but does not express any value
>    judgement about excessive variability.
> C. SpecGL enumerates the DoV and discourages variability beyond
>    a certain objective criterion.

David,

	Nice rendering of the options! A "clear" and "fair"
representation of opinions you disagree with. For completeness, let's
consider D:

D. SpecGL requires that it be possible to distinguish implementations
   that are conforming and non-conforming. Whatever DoV techniques are
   used by an implementation must comply with SpecGL requirements.
   Those requirements may or may not be specific to a certain DoV, as
   needed. If SpecGL has to require something specific to a DoV, that
   DoV must be defined/documented. If SpecGL has to require something
   specific to most currently known DoVs, a DoV framework may be
   appropriate as a part of SpecGL.

My intention is to arrive at a compact and precise spec, starting from
the most general requirements/properties of a "good spec" known today.
The opposite approach is to take all specific "bad things" from known
specs and demand that they do not exist in SpecGL-compliant specs,
generalizing where possible. In theory, both approaches converge. In
practice, it is often much more difficult to remove something from a
draft (even if it is redundant) than to add. Thus, I recommend
starting with general provisions and justifying every non-general
provision such as "modules".

> I think that we currently suffer at two levels, and SpecGL can help
> us in at least one of those levels. Those of us trying to write
> vendor- neutral conformance tests (my full-time job) suffer when the
> spec allows so much variability that there is little or no common
> ground across all conformant implementations. But we also suffer if
> we try to create an adaptable test suite, or to write
> vendor-specific tests that match the vendor's choices, because it is
> hard (and potentially subjective) to derive all the variability
> permitted by a typical W3C Rec of today. For example, if an
> implementer is allowed to implement one or more of four independent
> (parallel) modules, we can have an adaptable test suite that tests
> each module independently only if all parties agree on what the
> modules are. Thus, the identification of the modules is just as much
> a requirement on the specs as any other substantive requirements on
> behavior. Explicitly defining the modules as (magic word) "modules"
> sets conformance expectations at a level that enhances the ability
> of independent testing labs to conduct their own test and achieve
> consistent results. The specs can and should embody techniques that
> will ward off "we conform"/"no you don't" arguments by vendors and
> frustrated users. Please note that this benefit (someone other than
> the vendor can test whether the implementation conforms) leads to
> the benefit of interoperability, but it's separate. Making choice C
> above addresses interoperability.

You have outlined a real problem. You have proposed a solution
(require that something is labeled as a "module" in the spec). I do
not believe your solution is general enough to be in SpecGL. SpecGL
should require that "independent testing labs [should be able] to
conduct their own test and achieve consistent results". Such a
requirement can probably be expressed using general terms/rules.
Whether to label a part of a spec as a "module", should be left to
spec authors to decide.

> I think we all agree that the SpecGL should define a complete boundary
> between conforming and non-conforming implementations. I think a spec
> should be able to express requirements of the form "If you choose to
> implement X, you must implement it this way." This means that there is
> such a thing as constrained discretion, but it also means there is
> variability among implementations. QAWG will facilitate the creation of
> vendor-neutral test suites and will have to deal with constrained
> discretion. To maximize the benefits of open systems, the test suites
> and specs should facilitate testing by neutral third parties.

Agreed.

> Hence there is a need for clear delineation of variability
> techniques in use: modules implemented or not

What if a spec does not use anything that meets a "module" definition?
For example, I cannot think of many good "modules" within HTTP.

> which choice on discretionary choices,

That's an implementation choice, not related to spec, I guess. Spec
authors should have the right to have "discretionary choices" in the
spec, as long as discretional behavior is documented.

> deprecated items accepted (under their old meaning) or causing an
> error, and so forth.

This should be a part of the spec, but is not related to modules.

> It is right and proper for the SpecGL to impose formalisms on the WG
> writing the spec, so that multiple implementations can be tested by
> an independent lab despite the vendors taking advantage of allowable
> variability.

Agreed. It is not right, IMO, to impose formalism that is not general
enough.

> I suppose that any given test case in a suite can be characterized
> as applicable to implementations only if certain variability
> criteria are met. Example: this test only applies to Xblah consumers
> that implement the audio module, at Level 2, accept deprecated
> content from Version 3.0 and higher, chose to ignore (rather than
> error on) negative values of the foo parameter, and have splatness
> set on. This test can be a valid conformance test if a particular
> outcome is required under those conditions. But if the spec doesn't
> "package" the variability using the DoV techniques or something like
> them, the test suite would be very chaotic.

> SpecGL can save us from this chaos.

Perhaps, but "modules" cannot. I can rephrase the above using
"modules" terminology without any material change to the testability
of a spec. SpecGL should address a general problem (excessive
variability), if possible, and should not propose a specific
one-size-fits-all solution (call it a module!).

> Notice however, the necessity that the test writers know whether *or
> not* each DoV might be used. Saying explicitly "there are no
> modules" is better than saying nothing about modules and leaving the
> reader to guess.

If the spec in question is SpecGL-compliant, then it does not have
modules if modules are not specifically introduced/defined. And
vice-versa, if the spec is SpecGL-compliant and uses modules, then
they must be specifically introduced/defined. This goes from general
rules/requirements, not specific to DoV.

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Monday, 7 October 2002 17:23:14 UTC