Re: new Guidelines drafts posted

On Sun, 5 Jan 2003, David Marston/Cambridge/IBM wrote:

> >> Guideline 2. Identify what needs to conform and how.
>
> >What negative impact will removing this guideline from SpecGL have?
>
> This GL arose because a spec that defines the behavior of a consumer
> of the described XML may be seen as imposing some constraints on the
> producers of that XML.

Would not scope declaration address this already? I still do not see
any harm (or benefit) if somebody tries to apply a spec to an
out-of-scope device.

> >> Checkpoint 3.1. Specify any universal requirements for minimum
> >> functionality. [Priority 1]
>
> >I also question the priority level assigned to this checkpoint.
> >Again, given a good conformance clause, having "minimum
> >functionality" documented is a nice-to-have not must-have.
>
> Think about creating a test suite. Think about vendors bragging
> that their products are conformant. Think about interoperability.
> If this checkpoint is met, then the WG can identify a set of test
> cases that all conformant products must pass.

That would be nice but is not essential in any way.

Why make this a special case? Conformance clause must state what it
takes to become conformant. There is no need to talk about "minimum
functionality" in SpecGL. If a spec has minimum functionality, its
conformance clause would define it (explicitly or not). Thus, any
conformance policy conformant with SpecGL already ensures that "the WG
can identify a set of test cases that all conformant products must
pass". No need for a special checkpoint.

> >> Checkpoint 4.3. If profiles are chosen, define their relationships
> >> and interaction with other dimensions of variability. [Priority 2]
> >> To fulfill this checkpoint, a specification MUST identify all the
> >> relations and interactions between profiles and the other
> >> dimensions of variability. It is not applicable if profiles are not
> >> used.
>
> >This checkpoint is not testable. The authors may claim that all
> >interactions are identified, but it is not possible, in general, to
> >verify that the authors did not miss any implicit interactions.
>
> I wouldn't want to abandon this checkpoint just because it could be
> difficult. If the WG overlooked an interaction in Working Drafts,
> it should be revealed when developers try to implement the spec and
> when testers try to write test cases. My advice to the person trying
> to assess the spec on this checkpoint is to review as follows:
> For each profile, check whether the profile requires particular
> choices for any discretionary items, check whether the profile has
> levels, check whether the profile requires particular levels of any
> common features, check whether the profile constrains extensibility,
> [and so on].

Still, this checkpoint is not testable. The review you propose will
not be able to confirm that all "relations and interactions" are
identified. If you argue that the checkpoint should be left, then it
should be marked explicitly as not testable and, perhaps, marked as
non-normative.

> >> Checkpoint 5.1. If modules are chosen, indicate any mandatory
> >> conditions or constraints on their usage. [Priority 1]
>
> >This checkpoint is not testable.
>
> Unlike profiles, modules are always completely enumerated in the
> spec. This guideline is simply allowing for the possibility that
> a particular module could be a "core module" required of all
> products in a class. That seems testable to me. Maybe this
> guideline needs a rewrite.

The checkpoint, in its current shape, talks about identifying all
mandatory conditions or constraints, not about identifying all
modules. It is not possible to test that all implicit (but mandatory)
conditions or constraints are identified. If we are talking about
explicit conditions or constraints only, then, again, they are
identified by definition.

> >> Checkpoint 7.1. Identify each deprecated feature.
>
> >This checkpoint is not testable. See comments for checkpoint 4.3
> >above.
>
> In other words, you think a feature could be *implicitly*
> deprecated? How would that occur? Would it make it harder to write
> test cases for that spec, thus violating the guideline?

A feature could be implicitly deprecated if a spec says something like
"all features using FOO are now deprecated" or something like "all
features requiring compliance with spec BAR are now deprecated". Also,
again, if we are talking about explicitly deprecated features only,
then they are identified by definition! So, as many other checkpoints,
this checkpoint is either not testable or completely redundant.

> >> Checkpoint 8.2. For implementation dependencies, address the
> >> allowable differences between implementations [Priority 1]
>
> >This checkpoint is not testable.
>
> A non-ideal way to test the spec is to attempt to code an
> implementation, and while coding keep a log of all instances
> where the spec left externally-detectable behavior unspecified.
> (Sounds like current practice.)

Testing spec conformance to SpecGL by implementing the spec is not
practical for QA. Ideally, specs should be tested for SpecGL
conformance before people start wasting their time writing
implementations.


Thank you,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Monday, 6 January 2003 11:51:36 UTC