Re: lack of testability definition

Here is another view of testability - this came from a different NIST 
Laboratory - one that focuses on manufacturing.

Testability -- [1] The quality of a specification that enables
meaningful, repeatable, objective assessment of conformity to the
requirements therein. [2] The quality of a system that enables
meaningful, repeatable, objective assessment of conformity to
specified requirements.
--lynne


At 06:23 PM 10/27/2003, Alex Rousskov wrote:


>I owe Patrick Curran a blob on "lack of testability definition" for
>TestGL. Here it is:
>
>         QA Framework in general and this document in particular
>         rely on an an undefined term "testable" when defining and
>         using key concepts such as test assertions. The lack of a
>         definition for "testable" is not an oversight. The
>         discussion and examples below explain why there is
>         no good definition that is as general as QA Framework
>         or TestGL scope.
>
>         Informally, "boolean condition X is testable" usually means
>         "there exist a procedure to determine the truthfulness of X".
>         A more specific (and less general) wording may require the
>         "procedure" to be "finite" or even "affordable". The problem
>         with this informal approach is that for most practical
>         purposes related to computers, it is impossible to determine
>         truthfulness by following a procedure. The best we can do is to
>         attain a high level of confidence that X is true. Thus,
>         the informal definition that most people would agree with
>         may read "there exist a finite procedure to attain a high
>         level of confidence that X is true". What level of
>         confidence is "high enough" is, essentially, up to the
>         tester to define for a given environment. The cost
>         of the testing procedure and the cost of a false answer,
>         among other factors, would affect that environment-specific
>         definition.
>
>         To summarize, our inability to define "testable" lies
>         in the probabilistic nature of computer-related tests
>         and the fact that no general definition of "good
>         probability" or "high enough confidence" can exist.
>         The nature of the problem makes such definition
>         very subjective and/or domain-specific.
>
>         For example, a "software system MUST accept invalid
>         input" behavioral requirement can be tested using a black-box
>         technique with all sorts of invalid inputs and under
>         different conditions. However, there is no guarantee
>         that the invalid input that crashes the system was
>         tried. In most interesting cases the number of
>         invalid inputs and/or conditions is infinite and not
>         even countable. Whether, say, 99% confidence level would
>         be acceptable for concluding conformance in this case
>         would depend on things like software domain (isolated home PC
>         versus nuclear plant). Not to mention that there are
>         many ways to calculate that confidence level.
>
>         A less obvious example is a "document syntax conforms to
>         specification X" formatting requirement. Even if
>         specification X is written in a formal language, verifying
>         conformance of large documents would be impractical without
>         the use of automated validators. Thus, one would essentially
>         reformulate the above requirement into something like
>         "a correct document validator states that the
>         document syntax conforms to specification X". The first
>         sub-requirement ("the validator is correct") is equivalent
>         to the behavioral example discussed above.
>
>
>Please feel free to edit the above text or throw it away, of course.
>
>This submission is meant to complete AI-20031022-7, which is
>mis-stated as "come up with the definition for the term testable" at
>http://lists.w3.org/Archives/Public/www-qa-wg/2003Oct/0040.html
>AI-20031022-7 is due October 29.
>
>Thank you,
>
>Alex.

Received on Friday, 31 October 2003 09:16:41 UTC