- From: Alex Rousskov <rousskov@measurement-factory.com>
- Date: Fri, 30 Aug 2002 15:06:54 -0600 (MDT)
- To: www-qa@w3.org
Hi there, I have a few high-level comments on the published Specifications Guidelines. Overall, I find that the guidelines attempt to define/restrict/limit a lot more than they can or should. I also strongly disagree with an apparent intent to make every spec into an exciting novel to be enjoyed by humans on their way to work. IMHO, ideal specs should be succinct and formal, with no duplication of information, to the extent humanly possible. More Specific comments are below. "1.4 Terminology The keywords "MUST", "MUST NOT", ... will be used as defined in RFC 2119" The document does not use MUST, SHOULD, or MAY keywords. Or, at least, these words are not used in the way suggested by RFC 2119 (capitalized to make them look like keywords). "A specification is testable if there exists some finite cost-effective process with which a person or software can check that the requirement has been met." Virtually all specifications that define behavior based on inputs are not testable according to the above definition. There is no finite cost-effective process to enumerate all valid inputs. Yet such an enumeration would be required to check that implementation meets the requirement. The guidelines should either limit their scope to non-behavioral specifications or relax the definition of testability. "Guideline 1. Define Use Cases." I suggest that this is renamed to "Define Scope". Use cases often have a damaging effect because they limit the way spec authors interpret their spec (but not the way spec readers do). Scope is important and can be normative. Use cases are complimentary and should be used for illustrative (not normative) purposes only, if at all. "Checkpoint 1.1. Define the scope of the specification. ... The specification must clearly define what scenarios are in scope and what are out of scope in order to be interpreted, implemented, and tested correctly." Replace "clearly define" with "define" to make this checkpoint testable. In general, avoid adjectives in requirements as they have no value and may do harm. Search the guidelines for the word "clear" to find other places that need improvement (to start with). "Checkpoint 1.2. Include User Scenarios. [Priority 2]" "Checkpoint 1.3. Provide examples. [Priority 1]" These should have lower priority (priority 3?), IMO. Let specs concentrate on defining the standard. Let text books, articles, and such provide illustrations and examples. Ideal specs have non-normative statements only to make the text readable/understandable by smart humans, IMO. "The more numerous the examples, the better it is for the specifications comprehensibility." I strongly disagree. The more non-normative scenarios/examples there are, the higher the chance of a conflict between a normative statement and a non-normative example. Regardless of what the spec says about such conflicts and their resolution, they do lead to implementation bugs because some developers will look at the example and ignore or misinterpret the normative language. Ideally, there should be a single [normative] definition for everything in scope. No less, no more. "Checkpoint 1.4. Ensure that every test assertion is covered by an example. [Priority 3]" Why?! Satisfying this requirement will make protocol and similar specs huge without any added benefit. If, for example, a BNF specifies the protocol header syntax, why would anybody want to generate an example for everything that BNF covers?! "Checkpoint 1.5. Include an interpretation section. [Priority 2] It is hard to understand the formal descriptions of content without informative interpretations. The recent complex specifications like XML Schema [XML-SCHEMA] and XML Protocol have shown an necessity to have a "Primer" part or section to illustrate how to use the specification." This should have lower priority (priority 3?). If somebody finds it difficult to understand the formal descriptions, they should not rely on the specs to help them. See above for reasons not to mix normative and non-normative language and to avoid non-normative language. A short informal "Introduction" section per spec is fine. An informal interpretation for each formal description is very bad. "G 2. Identify what needs to conform and how. G 3. Address the use of profiles to divide the technology. G 4. Address the use of modules to divide the technology." I would shorten the three guidelines above into a single guideline with 3-5 paragraph of text addressing a single (and the only, IMO) important issue: "G 2. Define conformance". That is, the spec MUST define what it means to conform to the spec. Given a device/document/software within the scope of the spec, it SHOULD be possible to test for conformance. That's all. There is no use for SpecGL to enumerate possible products that might be in scope of future specs. There is no use to talk at length about profiles and modules, their benefits and side-effects. Leave these arguments for the spec authors. You are not going to provide a clear one-size-fits-all answer any way. Demand only what is essential and universal: a conformance clause. In other words, the guidelines should mandate that the spec: - MUST define what is in scope - for things that are in scope, MUST define what it means to conform - SHOULD make it possible to test for conformance (note: not "to verify conformance") "Checkpoint 5.1. Make it clear ..." "Checkpoint 5.2. Make it clear ..." "Checkpoint 5.3. Make it clear ..." See about the use of adjectives in checkpoints. "Guideline 6. Clarify the relation between deprecated features and conformance." This guideline talks about one specific problem that some specs might encounter. I suggest that this guideline is removed because the other guidelines and checkpoints already cover what this guideline, essentially, talks about: the spec must define conformance. A good definition will cover deprecated features as needed, of course. "Guideline 7. Address the use of functional levels to divide the technology." Again, I see no practical use from inclusion of this guideline. If the specs have to use levels, they will. If they do not have to, they will not. This is not general enough for this guidelines document, IMO. "Checkpoint 8.4. Include a statement regarding consistent handling of a discretionary item within an implementation. [Priority 2] The effect of each individual discretionary item should be consistent within a single implementation. For example, a browser's rendering of a XSL-FO (XSL Formatting Object) should be the same for every invocation regardless of the document instance." Why on Earth would these guidelines require something specific like that? Why handling of a discretionary item within an implementation should be consistent? Care to define an implementation in this context? Is browser X that is user-configured to do FOO the same implementation as browser X that is user-configured to do BAR? Care to define an invocation in this context? Can browser display cached documents differently from first-hand responses? Etc., etc. This is just too low-level to provide a general guideline, IMO. "Guideline 9. Allow extensions or NOT!" This guideline sounds cool, but should be removed because it does not cover any new ground, IMO: If the spec explicitly mentions an extension, it MUST document compliant behavior just like for any other non-extension, due to other guidelines. The problem arises only when the spec implicitly allows something without defining a conformant reaction to that something. That problem should be addressed by more general guidelines: - Define compliant behavior for every conforming object that the spec is using (no "implicity", good "input") - Define compliant behavior for every non-conforming object that an implementation may encounter (no "implicity", bad "input") "Checkpoint 9.5. If extensions are allowed, register or publish them. [Priority 3]" In general, this is totally unnecessary and introduces too much overhead provided the good/bad input guidelines above are satisfied. "Checkpoint 10.5. Identify all dimensions of variability that are not used. [Priority 1] It must be possible for the reader to determine, from the conformance clause, if: there is only one class of product specified, there are no profiles, there is no division of the technology into modules, there are no deprecated features, there are no functional levels, there are no discretionary items, extensions are not allowed. " I think this is an unnecessary overkill. The scope and conformance sections MUST define scope and conformance. We should not require authors to define non-scope and non-conformance as well. Everything that is not covered by the scope, is out of scope. Everything that does not conform, does not. What good does it make to state *in the conformance clause* that, say, ``extensions are [not] allowed''?! The reader still has to read the specs to understand what an extension is anyway. While doing so, the reader will encounter specific conformance requirements for those extensions, if any. "Checkpoint 11.2. Provide specific wording of the claim. [Priority 3] A well-formed conformance claim includes: date of the claim, specification name, date and version, URI of the specification, conformance level satisfied, and information about the subject (that which is claiming conformance). Information about the subject refers to information such as, the name of the software or software component, version information, and operating environment." This is good but probably impractical. Even W3C "valid *ML" images do not (and cannot) satisfy this verbose format. "Guideline 12. Publish an Implementation Conformance Statement proforma." This is useful in some cases, but has nothing to do with the spec itself, IMO. I would remove this guideline. Also, keep in mind that it is impractical do provide a 1:1 mapping between an implementation and conformance claim. A vendor is not going to update their claim every time they do a minor software release, for example. They would just state that "product X complies with FOO" (see Checkpoint 12.2). "Checkpoint 13.2. Distinguish normative and informative text. [Priority 2]" This should have the highest priority (Priority 1?), IMO. The spec is no good if normative and informative statements are indistinguishable. "Finally, using an advanced schema for specification authoring allows for greater control over coverage of the specification in the Test Suite." Not really. It may make greater control over coverage easier to implement, that's all. Plain ASCII text allows for the same coverage control as *ML. "Guideline 15. Include test assertions." IMO, the specs should not include test assertions because there should be a single normative definition and there should be nothing else. This is similar to including informative examples and such. The rationale the guidelines give for including Guideline 15 are not really specific to the inclusion of test assertions into the spec: "Including test assertions as part of the specification facilitates and promotes the development of test materials." So lets include everything that facilitates and promotes then. Let's include translations to other languages, alternative formats, author commentary, code samples, etc. etc. "Test assertions (that are part of a specification) are publicly reviewed and agreed to" So can be test assertions published outside of the spec. "Detailed control over what parts of a specification an implementation supports, grouped by behavior, technical aspect (method/interface) or other conceptual grouping;" This should be provided by normative definitions, not test assertions. "Detailed reporting when testing an implementation using the test;" Unrelated to the inclusion of test assertions into the spec. Moreover, I can make an argument that including test assertions in the spec may cripple future test suites and even implementations! Spec implementors will see the assertions and make sure they write code that passes those test cases. Test suite developers will implement the same assertions that the implementors were looking at (at least they will start with them, many will not go beyond). As a result, the effectiveness of a test suite will be minimal. It is a well known rule of thumb that best testing is done by non-implementors. Including "standard" test assertions violates this rule in practice (in theory it would not matter, of course). "The test assertions of this Specification Guidelines document are found in the prioritized checkpoints. A checkpoint will contain at least one, and may contain multiple individual requirements. These requirements are the test assertions of this specification." Note that most of what you call "test assertions" cannot be tested with because they contain undefined adjectives like "clear". "test assertion: any sentence or equivalent assemblage of words and phrases that prescribes the behavior that must be obtained when a stimulus occurs under a certain set of conditions. (See also QA Glossary [QA-GLOSSARY].)" Can a test assertion be expressed in a formal language without words and phrases rather than in sentences of a human language? Thank you, Alex. -- | HTTP performance - Web Polygraph benchmark www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite | all of the above - PolyBox appliance
Received on Friday, 30 August 2002 17:06:55 UTC