Re: new Guidelines drafts posted

Alex,

Thanks for taking the time and making the extensive comments.  We will 
review all of your comments at next week's QAWG face-to-face.

Regards,
-Lofton.

At 12:46 AM 12/29/02 -0700, Alex Rousskov wrote:

>On Tue, 24 Dec 2002, Lofton Henderson wrote:
>
> > http://www.w3.org/QA/WG/2002/12/qaframe-spec-20021220.html
> > ...
> > These versions are the base documents for discussion at the QA WG/IG
> > face-to-face meeting next month (6-8 January 2003) in Seattle [1].
> > We welcome comments, on this list.  If you comment before the
> > Seattle meeting, we will be able to discuss and resolve your
> > comments in time to potentially affect the Last Call versions.
>
>Here are a few comments. Original SpecGL text is quoted using a ">"
>prefix.
>
>
>         > * a statement of how to fulfill the checkpoint
>
>All these statements should be reworded to remove "To fulfill this
>checkpoint," phrases that only pollute requirements.
>
>         > Checkpoint 1.1. Define the scope of the specification.
>         >
>         > To fulfill this checkpoint, a specification MUST define what
>         > scenarios are in scope and SHOULD identify out of scope scenarios.
>
>The above wording is not precise enough. It is not clear whether ALL or just
>some in- and out-of-scope scenarios must/should be defined. Is one scenario
>enough? This gets pretty messy because "scenario" (or use case) is not a
>well-defined term.
>
>If ALL scenarios must be defined, then the checkpoint becomes untestable
>because it is impossible, under general assumptions, to know whether all
>scenarios have been covered by the scope definition.
>
>Defining out of scope scenarios should be a MAY. There is usually no harm
>whatsoever in not explicitly listing out-of-scope scenarios. In fact,
>the presence of out-of-scope scenarios often indicates shortcuts that the
>authors had to take: "security considerations are currently out of scope".
>
>To solve the above problems, I would suggest that the above wording is
>rephrased as follows:
>
>         Checkpoint 1.1. Define the scope of the specification.
>
>         A specification MUST define its scope.
>         A specification MAY define what is outside of its scope.
>
>
>         > Checkpoint 1.2. Provide Usage Scenarios. [Priority 2]
>
>         > To fulfill this checkpoint, a specification MUST include or link
>         > to Usage Scenarios, SHOULD have an extensive list of the usage
>         > scenarios that the authors have in mind
>
>The difference between "usage scenarios" in checkpoint 1.2 and
>"scenarios in scope" in 1.1 is not clear. If there is no difference,
>than checkpoint 1.2 is identical to 1.1. If there is a difference it
>should be explained. Note that the problem does not exist if the
>alternative wording for checkpoint 1.1 is used; in that case, 1.1 may be
>the same as 1.2 (depending on how the spec authors choose to define
>"scope"), but that's OK.
>
>Also, what is the difference between "includ[ing] Usage Scenarios" and
>"hav[ing] a list of the usage scenarios"? Isn't that the same thing,
>except the latter is more specific (too specific, perhaps).
>
>To solve the above problems, I would suggest that the above wording is
>rephrased as follows:
>
>         Checkpoint 1.2. Provide Usage Scenarios. [Priority 2]
>
>         A specification MUST document at least one Usage Scenario.
>
>Non-normative text should probably say that "documented Usage Scenarious
>do not define or limit the scope of the specification unless noted
>otherwise." We should make it clear that Usage Scenarios are their for
>illustrative purposes only and, in general, cannot cover all valid uses.
>In other words, Usage Scenarios are non-normative _examples_.
>
>         > Rationale: Usage scenarious help to assess what features are
>         > missing and what features are superfluous in the specification.
>
>Usage scenarios on their own cannot detect missing or extra features.
>Usage scenarios just illustrate scope or applicability of the
>specification. The above can be reworded to make more practical sense:
>
>         Rationale: documenting usage scenarios helps specification
>         authors to assess specification scope
>
>
>         > Checkpoint 1.3. Provide examples. [Priority 1]
>
>         > To fulfill this checkpoint, a specification MUST include or link to
>         > at least one example.
>
>An example of what? Are we talking about usage examples/scenarios (then
>it is a repetition of 1.2), something else?
>
>Here and elsewhere, replace "include or link" and "include or reference"
>with "document" or "specify". It should be up to the authors whether
>linking is an acceptable documentation/specification method (and the
>choice will probably depend on the context).
>
>
>         > Checkpoint 1.4. Provide an example for each section that is
>         > difficult to understand . [Priority 2]
>
>Remove this checkpoint: it is not testable for several reasons and it
>restricts authors in how complexity should be addressed. The intent here
>is clear: we want specifications to be easy to understand. We cannot
>demand that via a testable checkpoint though. It should be our hope that
>other [testable] checkpoints and authors goodwill will lead to specs
>that are easier to understand.
>
>         > Guideline 2. Identify what needs to conform and how.
>
>What negative impact will removing this guideline from SpecGL have? Its only
>contribution is an attempt to standardize spec categories (without defining
>those proposed categories!). Instead of having a special guideline, can we
>just suggest that proposed categories are used when "conformance" is defined
>in the specification (subject of Guidelines 3 and 10)?
>
>
>         > Guideline 10, "conformance clause" is related to this guideline.
>         > This Guideline 3 focuses on the establishment and scope of
>         > definition of a conformance policy, while Guideline 10 focuses on
>         > where and how to document it. That is, the verification of these
>         > checkpoints will require looking at the Conformance Clause.
>
>There is so much overlap between GL3 and GL10 that I have to question
>the rationale for keeping those two guidelines separate. I can
>understand "where and how to document" focus. I am having trouble to
>interpret "the establishment and scope of definition" focus. Does not
>documentation imply establishment? Does not [good] documentation of
>conformance policy implies its definition? I would suggest that GL3 and
>GL10 are merged into one guideline:
>
>         GL?: Define conformance.
>
>
>         > Checkpoint 3.1. Specify any universal requirements for minimum
>         > functionality. [Priority 1]
>         >
>         > To fulfill this checkpoint, a specification MUST include a
>         > normative section detailing any universal requirements for minimum
>         > functionality. It is not applicable if there are not any universal
>         > requirements.
>
>This checkpoint seems to be too specific/narrow and not practically
>useful given a good conformance policy. That is, in complex specs with
>many subjects, there cannot be a universal minimum functionality
>requirement. In simpler specs the minimum functionality should be
>obvious from the conformance definition/rules. In many specs, it would
>be impractical to confine the description to one section.
>
>I also question the priority level assigned to this checkpoint. Again,
>given a good conformance clause, having "minimum functionality"
>documented is a nice-to-have not must-have.
>
>If you think the checkpoint is still useful enough, I would suggest to
>reword it for clarity:
>
>         Checkpoint 3.1. Specify universal requirements for minimum
>         functionality. [Priority 3]
>
>         If specification has several subjects, and if all specification
>         subjects have common minimum functionality, then specification
>         SHOULD explicitly document that minimum.
>
>
>         > Checkpoint 3.3. Distinguish requirements from product-specific
>         > extra features [Priority 1]
>         >
>         > To full this checkpoint, a specification MUST state in its
>         > conformance section all facets of the requirements where the
>         > required features represent the maximum allowable capabilities.
>
>To tell you the truth, I do not understand what this checkpoint and
>accompanying rationale clause are trying to accomplish. The example in
>the rationale is an example of an extension (support for a graphical
>resolution other than those explicitly required to support). On the
>other hand, the rational says "implementations may be allowed to exceed
>the specified capabilities in ways other than having extensions". If I
>am not the only one confused by this checkpoint, perhaps the author
>could rephrase or clarify the intent.
>
>Also, it is often not practical to list all applicable "facets"
>(extensions or whatever) in the conformance section as there may be
>hundreds of them.
>
>         > If profiles are used (see Guideline 4), state whether ...
>
>The above paragraph has two identical "If levels are used ..."
>sentences.
>
>
>         > Checkpoint 3.4. If special conformance terms are used, include a
>         > definition in the specification. [Priority 1]
>         >
>         > To fulfill this checkpoint, a specification MUST either
>         > exclusively use conformance terms as defined in this document or
>         > define any other conformance terms used in it and reference them
>         > from the conformance clause.
>
>Consider more general:
>
>         Checkpoint 3.4. Conformance terms must be defined.
>
>         Conformance terms MUST be defined.
>
>Checkpoint comments may explain that external definitions are allowed
>(of course) as long as their sources are explicitly cited. The current
>checkpoint wording gives SpecGL a special status (only its definitions
>can be used without re-defining them) and requires copying definitions
>from any other spec.
>
>
>         > Checkpoint 3.5. Justify any usage of a dimension of variability by
>         > reference to usage scenarios and requirements [Priority 1]
>
>         > To fulfill this checkpoint, a specification MUST provide a
>         > justification for each Dimension of Variability the specification
>         > uses by reference to usage scenarios and requirements.
>
>Let's not suggest a specific and rather limited solution/technique.
>Consider more general:
>
>         Checkpoint 3.5. Justify used dimensions of variability. [Priority 1]
>
>         If a specification uses Dimensions of Variability,
>         it MUST justify each Dimension it uses.
>
>Same comment for Checkpoint 9.2, BTW.
>
>
>         > Checkpoint 4.3. If profiles are chosen, define their relationships
>         > and interaction with other dimensions of variability. [Priority 2]
>
>         > To fulfill this checkpoint, a specification MUST identify all the
>         > relations and interactions between profiles and the other
>         > dimensions of variability. It is not applicable if profiles are not
>         > used.
>
>This checkpoint is not testable. The authors may claim that all
>interactions are identified, but it is not possible, in general, to
>verify that the authors did not miss any implicit interactions. If we
>are only concerned about explicit interactions, then all explicitly
>defined interactions are, of course, "identified" (i.e., any spec would
>satisfy the above checkpoint).
>
>         > Checkpoint 5.1. If modules are chosen, indicate any mandatory
>         > conditions or constraints on their usage. [Priority 1]
>         >
>         > To fulfill this checkpoint, a specification MUST document any
>         > mandatory conditions or constraints on their usage.
>
>This checkpoint is not testable. See comments for checkpoint 4.3 above.
>
>         > Checkpoint 7.1. Identify each deprecated feature. [Priority 1]
>         >
>         > To fulfill this checkpoint, a specification MUST identify in a
>         > normative section each deprecated feature. It is not applicable if
>         > there is no deprecated feature.
>
>This checkpoint is not testable. See comments for checkpoint 4.3 above.
>
>         > Checkpoint 7.4. Include examples to illustrate how to avoid using
>         > deprecated features. [Priority 3]
>         >
>         > To fulfill this checkpoint, a specification MUST provide an
>         > example for each deprecated feature showing how to avoid using it.
>         > It is not applicable if there is no deprecated features.
>
>This should be SHOULD-level requirement because it is talking about
>examples, which cannot be MUSTs because some specifications (i.e., those
>written in formal languages) do not need them and cannot have them.
>
>Also, we may want to keep inmind that requiring such examples often does not
>help much. For example, recent HTML specs often would say something like this:
>"<foo> is depricated, use style sheets instead" which is kind of an example,
>but may not specific enough for a non-guru to actually avoid using <foo>.
>
>         > Checkpoint 8.1. State the circumstances for when discretionary
>         > items are allowed [Priority 1]
>         >
>         > To fulfill this checkpoint, a specification MUST indicate the
>         > rationale for the discretionary items by providing a reference or
>         > link to its use cases and/or project requirements and SHOULD
>         > identify by labeling all discretionary items. It is not applicable
>         > for specifications that don't have discretionary items.
>
>SpecGL should not prescribe a specific technique ("by providing a reference or
>link" and "by labeling").
>
>The SHOULD part is not testable. See comments for checkpoint 4.3 above.
>
>
>         > Checkpoint 8.2. For implementation dependencies, address the
>         > allowable differences between implementations [Priority 1]
>
>         > To fulfill this checkpoint, a specification MUST describe any
>         > permitted variations or constraints for how the implementation
>         > dependency is realized by implementations.
>
>This checkpoint is not testable. See comments for checkpoint 4.3 above.
>To summarize, pretty much all checkpoints of the "MUST idedtify all X"
>form, where X is introduced by the specification itself are either not
>testable (if X can be introduced implicitly) or self-referential (if X
>must be defined to exist).
>
>
>
>         > Checkpoint 8.3. Indicate any constraints on the number of
>         > choices/options that can be implemented [Priority 2]
>         >
>         > To fulfill this checkpoint, a specification MUST indicate whether
>         > zero, exactly one, or a multiple of choices/options are allowed to
>         > be implemented. It is not applicable for specifications that don't
>         > use discretionary items or for implementation dependent values 
> among
>         > them.
>
>What does "them" refer to? Also, let's not introduce any magic constants
>like "zero" or "one":
>
>         Checkpoint 8.3. Indicate any constraints on the number of
>         choices/options that can be implemented [Priority 2]
>
>         A specification MUST document how many choices/options a single
>         implementation can support at a time.
>
>I also question the necessity of this checkpoint. Why is it important?  There
>is no rationale clause to explain.  Should we demand documentation of options
>interaction instead? Most comments for checkpoint 8.4 below apply here as well
>("single implementation", "at a time").
>
>
>         > Checkpoint 8.4. Promote consistent handling of discretionary
>         > choices. [Priority 2]
>         >
>         > To fulfill this checkpoint, the specification MUST state that
>         > given identical conditions, the effect of a discretionary choice is
>         > consistent within a single implementation.
>         >
>         > Rationale. Users have an expectation of what to expect and should
>         > be able to count on getting the same results under the same
>         > conditions with a given implementation.
>
>This checkpoint should be removed. It introduces many undefined and/or
>specific (not general enough terms):
>
>         - "identical conditions" (is current time or random number generator
>           state included in the "condition"?)
>
>         - "effect of a choice" (effect on user? on log files? on CPU
>           utilization? temperature in the room?)
>
>         - "consistent effect" (consistent with what? user expectations or
>           design rationale? why not identical?)
>
>         - "single implementation" (if a user reconfigures a product
>           run-time, is it the same implementation or a different one?
>           what if the implementation gets automatic updates, is
>           self-configurable, or uses true entropy source?)
>
>         - "same result" (see "effect of a choice")
>
>If somebody believes the checkpoint must stay, please provide more
>convincing arguments why every good specification on Earth that uses
>discretionary items MUST state "given identical conditions, the effect
>of a discretionary choice is consistent within a single implementation"
>and close the above terminology holes. At the very least, MUST should be
>replaced with MAY and priority should be decreased.
>
>
>         > For example, a specification that allows private extensions (e.g.,
>         > proprietary) is highly likely to impede interoperability, whereas a
>         > specification than permits only registered extensions partially
>         > mitigates the negative impacts.
>
>This is a classic example of wishful thinking. Even if the above thesis
>was true in theory (public is always better than private), in real
>world, it is usually impossible to enforce registration constraints and,
>hence, private extensions are at least as easy to introduce as public
>ones. In other words, when you talk about proprietary interoperability,
>it really does not matter what specification permits because private
>extensions usually do not have to be compliant to "work". I would
>suggest that the above example is removed.
>
>
>         > Checkpoint 9.4. Use a standard mechanism to define the extension.
>         > [Priority 3]
>         >
>         > To fulfill this checkpoint, a specification MUST provide a
>         > standard way of defining the extension. It is not applicable if
>         > extensions are not allowed.
>
>What is a "standard mechanism/way"? Which standard does this checkpoint
>refer to?
>
>         > Checkpoint 9.6. Require implementations to include a way to "turn
>         > off" extensions. [Priority 3]
>         >
>         > To fulfill this checkpoint, a specification MUST indicate via
>         > conformance requirements that implementations provide a mode under
>         > which they produce only conforming content. This checkpoint is not
>         > applicable if extensions are not allowed.
>
>This checkpoint is probably too narrow and should be removed. To start
>with, many implementations do not produce any content. Also, if
>extensions do not hurt interoperability, it is not clear why a
>conformant implementation MUST be able to disable them. Presence of
>extensions does not imply conformance violations.
>
>
>Checkpoint 10.2 is not testable. See comments for checkpoint 4.3 above.
>
>
>AFAIK, IETF requires standards to have Security Considerations sections in
>addition to conformance sections. Should W3C do the same?
>
>
>Finally, somebody needs to spellcheck SpecGL :-).
>
>
>HTH,
>
>Alex.
>
>--
>                             | HTTP performance - Web Polygraph benchmark
>www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
>                             | all of the above - PolyBox appliance

Received on Thursday, 2 January 2003 16:04:14 UTC