Re: How do we test a specification?

I think we may need to define or articulate as specifically as possible what is 
meant by "testing a specification".  Is it meant that all requirements 
contained in the specification can be deterministically proven to be true/false 
(ref: definition of "testability" in QA Spec GL [1]) when applied to 
implementations of the specification?  This would be reading the 
sentence "testing (implementations of) the specification".   Or, "parsing the 
phrase "testing the specifcation" in a different way (using "specification" as 
a direct object of the verb "test" in the sentence) is it meant that the 
specification is "well written" according (for example) to SpecGL 
requirements?   Or, does it mean that two implementations of the same 
specification will behave the same way when measured against the specification 
requirements?  

Finally: Is the interpretation of "testing a specification" dependent upon the 
nature of the specification requirements?  

For example, in terms of "testing the WCAG2.0 specification", the success 
criteria must be "testable" in specific technologies, although the success 
criteria are defined in a technology-neutral way.  This is becuase the concepts 
of "accessibility" are only evident when those concepts can be practically 
demonstrated via techniques using specific technologies.  The same concept may 
have different demonstrations (ways in which the concept can be satisfied) 
depending on which technologies are being employed.   So WCAG tests would 
presumably measure the success of such demonstrations.   Care must be taken in 
such tests to demonstrate accessibility present in a technology and not test 
the basic features of the technology supporting the accessibility 
demonstration.  Also, such tests may invariably involve human testing in 
addition to machine testing to demonstrate the presence of accessibility 
concepts (to get "high confidence of truth of proposition).

However, in terms of "testing the CSS2.1 specification", there is a grammar 
(syntax) defined, as well as definition of media types and properties, so 
testability relates to correct parsing of the grammar as well as demonstrable 
support for at least one media type and property values according to the CSS2.1 
specification.  There is also language in CSS2.1 defining interoperability of
implementations in order to exit the Candidate Recommendation stage of the
W3C process. 

So, the meaning of "testing the WCAG2.0 specification" may be different than
the meaning of "testing the CSS2.1 specification"?

Thanks and best wishes
Tim Boland NIST 

[1]: http://www.w3.org/TR/2005/REC-qaframe-spec-20050817/#glossary    


 Quoting Karl Dubost <karl@w3.org>:

> 
> Hi QA IG,
> 
> more and more often, I receive questions and/or comments from WGs  
> about the testability of their specifications. It's quite interesting  
> and shows part of the misunderstandings of what has to be done when  
> testing a specification. I would like to know if you have ideas about  
> the simple question of
> 
>      How do we test a specification?
> 
> Types of comment
> 
>      * Our language is defining semantics, then is not testable or we  
> can't create a test to prove implementation.
>      * Our language is defining a framework communications between  
> services, then it's dependent on the service and not testable.
> 
> 
> Do you have a set of guidelines to invite people to define the good  
> questions for creating Test Cases?
> 
> 
> 
> 
> Example:
> 
> I remember having this discussion about "blockquote", which is an  
> element in HTML 4.01 to markup quotes in a text.  Some people told  
> me, it's not testable, because you can't prove that the text inside  
> the element is really a quote. I agree with that but I think it's the  
> wrong question.
> How do we prove that blockquote is implementable? (or useful).
> 
> My suggestion was to prove that it was possible to implement by  
> creating for example a tool which extracts in HTML the quotes of a  
> text and then gives the possibility of a quote service. Extracting  
> the content of the quote and the attribute cite, is a kind of  
> implementation of blockquote.
> 
> So you prove that the element is useful by showing that it's usable  
> in the way it has been defined.
> 
> 
> 
> -- 
> Karl Dubost - http://www.w3.org/People/karl/
> W3C Conformance Manager
> *** Be Strict To Be Cool ***
> 
> 
> 
> 
>

Received on Monday, 31 October 2005 15:12:07 UTC