- From: Alex Rousskov <rousskov@measurement-factory.com>
- Date: Sun, 29 Dec 2002 00:46:07 -0700 (MST)
- To: www-qa@w3.org
On Tue, 24 Dec 2002, Lofton Henderson wrote:
> http://www.w3.org/QA/WG/2002/12/qaframe-spec-20021220.html
> ...
> These versions are the base documents for discussion at the QA WG/IG
> face-to-face meeting next month (6-8 January 2003) in Seattle [1].
> We welcome comments, on this list. If you comment before the
> Seattle meeting, we will be able to discuss and resolve your
> comments in time to potentially affect the Last Call versions.
Here are a few comments. Original SpecGL text is quoted using a ">"
prefix.
> * a statement of how to fulfill the checkpoint
All these statements should be reworded to remove "To fulfill this
checkpoint," phrases that only pollute requirements.
> Checkpoint 1.1. Define the scope of the specification.
>
> To fulfill this checkpoint, a specification MUST define what
> scenarios are in scope and SHOULD identify out of scope scenarios.
The above wording is not precise enough. It is not clear whether ALL or just
some in- and out-of-scope scenarios must/should be defined. Is one scenario
enough? This gets pretty messy because "scenario" (or use case) is not a
well-defined term.
If ALL scenarios must be defined, then the checkpoint becomes untestable
because it is impossible, under general assumptions, to know whether all
scenarios have been covered by the scope definition.
Defining out of scope scenarios should be a MAY. There is usually no harm
whatsoever in not explicitly listing out-of-scope scenarios. In fact,
the presence of out-of-scope scenarios often indicates shortcuts that the
authors had to take: "security considerations are currently out of scope".
To solve the above problems, I would suggest that the above wording is
rephrased as follows:
Checkpoint 1.1. Define the scope of the specification.
A specification MUST define its scope.
A specification MAY define what is outside of its scope.
> Checkpoint 1.2. Provide Usage Scenarios. [Priority 2]
> To fulfill this checkpoint, a specification MUST include or link
> to Usage Scenarios, SHOULD have an extensive list of the usage
> scenarios that the authors have in mind
The difference between "usage scenarios" in checkpoint 1.2 and
"scenarios in scope" in 1.1 is not clear. If there is no difference,
than checkpoint 1.2 is identical to 1.1. If there is a difference it
should be explained. Note that the problem does not exist if the
alternative wording for checkpoint 1.1 is used; in that case, 1.1 may be
the same as 1.2 (depending on how the spec authors choose to define
"scope"), but that's OK.
Also, what is the difference between "includ[ing] Usage Scenarios" and
"hav[ing] a list of the usage scenarios"? Isn't that the same thing,
except the latter is more specific (too specific, perhaps).
To solve the above problems, I would suggest that the above wording is
rephrased as follows:
Checkpoint 1.2. Provide Usage Scenarios. [Priority 2]
A specification MUST document at least one Usage Scenario.
Non-normative text should probably say that "documented Usage Scenarious
do not define or limit the scope of the specification unless noted
otherwise." We should make it clear that Usage Scenarios are their for
illustrative purposes only and, in general, cannot cover all valid uses.
In other words, Usage Scenarios are non-normative _examples_.
> Rationale: Usage scenarious help to assess what features are
> missing and what features are superfluous in the specification.
Usage scenarios on their own cannot detect missing or extra features.
Usage scenarios just illustrate scope or applicability of the
specification. The above can be reworded to make more practical sense:
Rationale: documenting usage scenarios helps specification
authors to assess specification scope
> Checkpoint 1.3. Provide examples. [Priority 1]
> To fulfill this checkpoint, a specification MUST include or link to
> at least one example.
An example of what? Are we talking about usage examples/scenarios (then
it is a repetition of 1.2), something else?
Here and elsewhere, replace "include or link" and "include or reference"
with "document" or "specify". It should be up to the authors whether
linking is an acceptable documentation/specification method (and the
choice will probably depend on the context).
> Checkpoint 1.4. Provide an example for each section that is
> difficult to understand . [Priority 2]
Remove this checkpoint: it is not testable for several reasons and it
restricts authors in how complexity should be addressed. The intent here
is clear: we want specifications to be easy to understand. We cannot
demand that via a testable checkpoint though. It should be our hope that
other [testable] checkpoints and authors goodwill will lead to specs
that are easier to understand.
> Guideline 2. Identify what needs to conform and how.
What negative impact will removing this guideline from SpecGL have? Its only
contribution is an attempt to standardize spec categories (without defining
those proposed categories!). Instead of having a special guideline, can we
just suggest that proposed categories are used when "conformance" is defined
in the specification (subject of Guidelines 3 and 10)?
> Guideline 10, "conformance clause" is related to this guideline.
> This Guideline 3 focuses on the establishment and scope of
> definition of a conformance policy, while Guideline 10 focuses on
> where and how to document it. That is, the verification of these
> checkpoints will require looking at the Conformance Clause.
There is so much overlap between GL3 and GL10 that I have to question
the rationale for keeping those two guidelines separate. I can
understand "where and how to document" focus. I am having trouble to
interpret "the establishment and scope of definition" focus. Does not
documentation imply establishment? Does not [good] documentation of
conformance policy implies its definition? I would suggest that GL3 and
GL10 are merged into one guideline:
GL?: Define conformance.
> Checkpoint 3.1. Specify any universal requirements for minimum
> functionality. [Priority 1]
>
> To fulfill this checkpoint, a specification MUST include a
> normative section detailing any universal requirements for minimum
> functionality. It is not applicable if there are not any universal
> requirements.
This checkpoint seems to be too specific/narrow and not practically
useful given a good conformance policy. That is, in complex specs with
many subjects, there cannot be a universal minimum functionality
requirement. In simpler specs the minimum functionality should be
obvious from the conformance definition/rules. In many specs, it would
be impractical to confine the description to one section.
I also question the priority level assigned to this checkpoint. Again,
given a good conformance clause, having "minimum functionality"
documented is a nice-to-have not must-have.
If you think the checkpoint is still useful enough, I would suggest to
reword it for clarity:
Checkpoint 3.1. Specify universal requirements for minimum
functionality. [Priority 3]
If specification has several subjects, and if all specification
subjects have common minimum functionality, then specification
SHOULD explicitly document that minimum.
> Checkpoint 3.3. Distinguish requirements from product-specific
> extra features [Priority 1]
>
> To full this checkpoint, a specification MUST state in its
> conformance section all facets of the requirements where the
> required features represent the maximum allowable capabilities.
To tell you the truth, I do not understand what this checkpoint and
accompanying rationale clause are trying to accomplish. The example in
the rationale is an example of an extension (support for a graphical
resolution other than those explicitly required to support). On the
other hand, the rational says "implementations may be allowed to exceed
the specified capabilities in ways other than having extensions". If I
am not the only one confused by this checkpoint, perhaps the author
could rephrase or clarify the intent.
Also, it is often not practical to list all applicable "facets"
(extensions or whatever) in the conformance section as there may be
hundreds of them.
> If profiles are used (see Guideline 4), state whether ...
The above paragraph has two identical "If levels are used ..."
sentences.
> Checkpoint 3.4. If special conformance terms are used, include a
> definition in the specification. [Priority 1]
>
> To fulfill this checkpoint, a specification MUST either
> exclusively use conformance terms as defined in this document or
> define any other conformance terms used in it and reference them
> from the conformance clause.
Consider more general:
Checkpoint 3.4. Conformance terms must be defined.
Conformance terms MUST be defined.
Checkpoint comments may explain that external definitions are allowed
(of course) as long as their sources are explicitly cited. The current
checkpoint wording gives SpecGL a special status (only its definitions
can be used without re-defining them) and requires copying definitions
from any other spec.
> Checkpoint 3.5. Justify any usage of a dimension of variability by
> reference to usage scenarios and requirements [Priority 1]
> To fulfill this checkpoint, a specification MUST provide a
> justification for each Dimension of Variability the specification
> uses by reference to usage scenarios and requirements.
Let's not suggest a specific and rather limited solution/technique.
Consider more general:
Checkpoint 3.5. Justify used dimensions of variability. [Priority 1]
If a specification uses Dimensions of Variability,
it MUST justify each Dimension it uses.
Same comment for Checkpoint 9.2, BTW.
> Checkpoint 4.3. If profiles are chosen, define their relationships
> and interaction with other dimensions of variability. [Priority 2]
> To fulfill this checkpoint, a specification MUST identify all the
> relations and interactions between profiles and the other
> dimensions of variability. It is not applicable if profiles are not
> used.
This checkpoint is not testable. The authors may claim that all
interactions are identified, but it is not possible, in general, to
verify that the authors did not miss any implicit interactions. If we
are only concerned about explicit interactions, then all explicitly
defined interactions are, of course, "identified" (i.e., any spec would
satisfy the above checkpoint).
> Checkpoint 5.1. If modules are chosen, indicate any mandatory
> conditions or constraints on their usage. [Priority 1]
>
> To fulfill this checkpoint, a specification MUST document any
> mandatory conditions or constraints on their usage.
This checkpoint is not testable. See comments for checkpoint 4.3 above.
> Checkpoint 7.1. Identify each deprecated feature. [Priority 1]
>
> To fulfill this checkpoint, a specification MUST identify in a
> normative section each deprecated feature. It is not applicable if
> there is no deprecated feature.
This checkpoint is not testable. See comments for checkpoint 4.3 above.
> Checkpoint 7.4. Include examples to illustrate how to avoid using
> deprecated features. [Priority 3]
>
> To fulfill this checkpoint, a specification MUST provide an
> example for each deprecated feature showing how to avoid using it.
> It is not applicable if there is no deprecated features.
This should be SHOULD-level requirement because it is talking about
examples, which cannot be MUSTs because some specifications (i.e., those
written in formal languages) do not need them and cannot have them.
Also, we may want to keep inmind that requiring such examples often does not
help much. For example, recent HTML specs often would say something like this:
"<foo> is depricated, use style sheets instead" which is kind of an example,
but may not specific enough for a non-guru to actually avoid using <foo>.
> Checkpoint 8.1. State the circumstances for when discretionary
> items are allowed [Priority 1]
>
> To fulfill this checkpoint, a specification MUST indicate the
> rationale for the discretionary items by providing a reference or
> link to its use cases and/or project requirements and SHOULD
> identify by labeling all discretionary items. It is not applicable
> for specifications that don't have discretionary items.
SpecGL should not prescribe a specific technique ("by providing a reference or
link" and "by labeling").
The SHOULD part is not testable. See comments for checkpoint 4.3 above.
> Checkpoint 8.2. For implementation dependencies, address the
> allowable differences between implementations [Priority 1]
> To fulfill this checkpoint, a specification MUST describe any
> permitted variations or constraints for how the implementation
> dependency is realized by implementations.
This checkpoint is not testable. See comments for checkpoint 4.3 above.
To summarize, pretty much all checkpoints of the "MUST idedtify all X"
form, where X is introduced by the specification itself are either not
testable (if X can be introduced implicitly) or self-referential (if X
must be defined to exist).
> Checkpoint 8.3. Indicate any constraints on the number of
> choices/options that can be implemented [Priority 2]
>
> To fulfill this checkpoint, a specification MUST indicate whether
> zero, exactly one, or a multiple of choices/options are allowed to
> be implemented. It is not applicable for specifications that don't
> use discretionary items or for implementation dependent values among
> them.
What does "them" refer to? Also, let's not introduce any magic constants
like "zero" or "one":
Checkpoint 8.3. Indicate any constraints on the number of
choices/options that can be implemented [Priority 2]
A specification MUST document how many choices/options a single
implementation can support at a time.
I also question the necessity of this checkpoint. Why is it important? There
is no rationale clause to explain. Should we demand documentation of options
interaction instead? Most comments for checkpoint 8.4 below apply here as well
("single implementation", "at a time").
> Checkpoint 8.4. Promote consistent handling of discretionary
> choices. [Priority 2]
>
> To fulfill this checkpoint, the specification MUST state that
> given identical conditions, the effect of a discretionary choice is
> consistent within a single implementation.
>
> Rationale. Users have an expectation of what to expect and should
> be able to count on getting the same results under the same
> conditions with a given implementation.
This checkpoint should be removed. It introduces many undefined and/or
specific (not general enough terms):
- "identical conditions" (is current time or random number generator
state included in the "condition"?)
- "effect of a choice" (effect on user? on log files? on CPU
utilization? temperature in the room?)
- "consistent effect" (consistent with what? user expectations or
design rationale? why not identical?)
- "single implementation" (if a user reconfigures a product
run-time, is it the same implementation or a different one?
what if the implementation gets automatic updates, is
self-configurable, or uses true entropy source?)
- "same result" (see "effect of a choice")
If somebody believes the checkpoint must stay, please provide more
convincing arguments why every good specification on Earth that uses
discretionary items MUST state "given identical conditions, the effect
of a discretionary choice is consistent within a single implementation"
and close the above terminology holes. At the very least, MUST should be
replaced with MAY and priority should be decreased.
> For example, a specification that allows private extensions (e.g.,
> proprietary) is highly likely to impede interoperability, whereas a
> specification than permits only registered extensions partially
> mitigates the negative impacts.
This is a classic example of wishful thinking. Even if the above thesis
was true in theory (public is always better than private), in real
world, it is usually impossible to enforce registration constraints and,
hence, private extensions are at least as easy to introduce as public
ones. In other words, when you talk about proprietary interoperability,
it really does not matter what specification permits because private
extensions usually do not have to be compliant to "work". I would
suggest that the above example is removed.
> Checkpoint 9.4. Use a standard mechanism to define the extension.
> [Priority 3]
>
> To fulfill this checkpoint, a specification MUST provide a
> standard way of defining the extension. It is not applicable if
> extensions are not allowed.
What is a "standard mechanism/way"? Which standard does this checkpoint
refer to?
> Checkpoint 9.6. Require implementations to include a way to "turn
> off" extensions. [Priority 3]
>
> To fulfill this checkpoint, a specification MUST indicate via
> conformance requirements that implementations provide a mode under
> which they produce only conforming content. This checkpoint is not
> applicable if extensions are not allowed.
This checkpoint is probably too narrow and should be removed. To start
with, many implementations do not produce any content. Also, if
extensions do not hurt interoperability, it is not clear why a
conformant implementation MUST be able to disable them. Presence of
extensions does not imply conformance violations.
Checkpoint 10.2 is not testable. See comments for checkpoint 4.3 above.
AFAIK, IETF requires standards to have Security Considerations sections in
addition to conformance sections. Should W3C do the same?
Finally, somebody needs to spellcheck SpecGL :-).
HTH,
Alex.
--
| HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
| all of the above - PolyBox appliance
Received on Sunday, 29 December 2002 02:46:52 UTC