RE: How do we test a specification?

As a pre-requisite to How do we test a specification... it the need to
have the requirements on which to base the tests.  The following are
some principles for writing these requirements.  Incidentally, some of
them apply to how to write a test (obviously #7 does not apply).  These
principles were taken from experience in writing requirements as well as
from various W3C WG documents on authoring tests.  

PRINCIPLES for Writing Conformance Requirements
General principles for creating conformance requirements:
1. Do restrict each conformance requirement to one atomic, simple
statement.
* Address one feature at a time.
* Keep each requirement as simple as possible. Multiple one-feature
requirements will be easier to test (and trace back should a test fail)
than one multi-part requirement.
* Order features in a natural progression, beginning with easiest. This
makes your document easier to read and later testing and trace-back more
sensible (see next item). If an implementation can't support an 'easy'
requirement, then it is unlikely to support a more complex one.
* Ensure traceability of requirements back to a specific part of the
specification. Each requirement must be directly traceable to wording in
the specification. Try to get as granular as possible, e.g., a specific
statement rather than a section.
2. Do make requirements technology neutral.
3. Do Not change functionality of the specification.
4. Do Not mix important terminologies.
* Use a Glossary agreed upon by all.
* Keep the balance of your text self-contained, with as few footnotes
and external references as practical.
* Choose either "shall" or "must" to signify requirement, use "should"
for recommendation (but not requirement) and "may" for neutral choice.
Don't mix sets of terms that assign different interpretations to the
same words.
5. Do constrain optionality and cardinality.
* Describe what features, values, attributes, etc. are to be measured
and what the range indications are for success or failure.
Example: ...occurs zero or more times
...required when <Person> used
6. Do indicate explicit dependencies and constraints.
7. Do Not state how to test.
8. Do Not rely upon formatting or context to convey intentions such as
mandatory or optional. For example, instead of employing italic or bold
face, use the English imperative "shall" or "must", whichever is chosen
as standard terminology.
Example: Not "..this feature is required," but rather, "..shall require
feature A31."


--regards
Lynne


-----Original Message-----
From: www-qa-request@w3.org [mailto:www-qa-request@w3.org] On Behalf Of
Karl Dubost
Sent: Thursday, October 27, 2005 4:52 PM
To: www-qa@w3.org
Subject: How do we test a specification?


Hi QA IG,

more and more often, I receive questions and/or comments from WGs  
about the testability of their specifications. It's quite interesting  
and shows part of the misunderstandings of what has to be done when  
testing a specification. I would like to know if you have ideas about  
the simple question of

     How do we test a specification?

Types of comment

     * Our language is defining semantics, then is not testable or we  
can't create a test to prove implementation.
     * Our language is defining a framework communications between  
services, then it's dependent on the service and not testable.


Do you have a set of guidelines to invite people to define the good  
questions for creating Test Cases?




Example:

I remember having this discussion about "blockquote", which is an  
element in HTML 4.01 to markup quotes in a text.  Some people told  
me, it's not testable, because you can't prove that the text inside  
the element is really a quote. I agree with that but I think it's the  
wrong question.
How do we prove that blockquote is implementable? (or useful).

My suggestion was to prove that it was possible to implement by  
creating for example a tool which extracts in HTML the quotes of a  
text and then gives the possibility of a quote service. Extracting  
the content of the quote and the attribute cite, is a kind of  
implementation of blockquote.

So you prove that the element is useful by showing that it's usable  
in the way it has been defined.



-- 
Karl Dubost - http://www.w3.org/People/karl/
W3C Conformance Manager
*** Be Strict To Be Cool ***

Received on Friday, 28 October 2005 13:32:33 UTC