Re: [www-qa] A good example of documenting for testability

David,

 From a testability perspective, I completely agree with your comments 
about the necessity of good documentation.  This example raises another 
interesting question, which goes beyond the issue of evaluating 
specification language for testability.

At 03:37 PM 9/7/01 -0400, David_Marston@lotus.com wrote:
>If you've been reading the W3C documents about QA, especially
>http://www.w3.org/2001/01/qa-ws/slides/qacirc.jpg
>(the concentric circles), know that an early focus will be on the
>documentation from which test suites are derived. The Recommendations
>and other normative documents must prefer explicit statements over
>silence,

Yes.

>and we have a good example at hand.
>
>I was just browsing through the Functions & Operators draft at
>http://www.w3.org/TR/xquery-operators
>and found this interesting issue on the issues list:
>
><quote>
>[...snip...]
>So I argue that if we want there to be no interaction with xml:lang, or
>if we want such an interaction to be legal but not required, or if we
>want it to be required, we ought to say explicitly what we want.
></quote>
>
>We need to encourage the idea that external/environmental factors must
>be addressed whenever they may result in variations of operation of
>the software.

I completely agree.  But the particular issue you quote may go deeper than 
its testability impact.  See below.

>In this example, consider how one would test sorting.
>Can a test author provide one worldwide standard for a correctly
>sorted result, or must the test harness check parameters of the
>environment and choose/generate a reference output to be matched? In
>the "legal but not required" scenario, the software developer must
>inform the world about whether they check the environment or not, and
>that bit of information must be fed into the test harness. It all
>starts with good documentation.
>.................David Marston

The "legal but not required" scenario has potential impact beyond 
testability.  As you point out, the testability issues could be dealt with 
by having the developer supply the proper information for the test 
harness.  But, conformance testing aside, do under-specified, 
discretionary, aspects of a standard have an impact on open 
interoperability; and, is it within the scope of a QA activity to address 
(comment on) such aspects of the specifications?

In my experience, there is an impact (a negative impact).  At the least, 
more discretion (optional features) can lead to more fragmentation in valid 
"flavors" of implementations, which can lead to more unpredictability in 
interchange transactions.  The seriousness of the impact on specific 
interchange transactions depends on how the discretionary specifications 
are written (are there "discovery" mechanisms for the user?  does the 
language have announcer features, so that the user can declare dependence 
upon availability of some particular optional capability?  etc).

My own bias is that discretionary behaviors should be avoided if possible, 
in the writing of the standards.  Whether it is within the scope of QA to 
address these aspects, I am uncertain.

Regards,
-Lofton.

*******************
Lofton Henderson
1919 Fourteenth St., #604
Boulder, CO   80302

Phone:  303-449-8728
Email:  lofton@rockynet.com
*******************

Received on Monday, 10 September 2001 13:11:20 UTC