Re: lack of testability definition

Lynn,

	I believe your quote is not another "view". I tried to distill
the definition to have only one undefined/subjective term left. What
you quote has at least three undefined/subjective terms -- it is just
a precursor or a less general version of my definition, IMO.

	It would be good to incorporate your quote as a perfect
example of why there can be no formal testability definition, only
fuzzy and custom wishlists.

Thank you,

Alex.


On Fri, 31 Oct 2003, Lynne Rosenthal wrote:

> Here is another view of testability - this came from a different NIST
> Laboratory - one that focuses on manufacturing.
>
> Testability -- [1] The quality of a specification that enables
> meaningful, repeatable, objective assessment of conformity to the
> requirements therein. [2] The quality of a system that enables
> meaningful, repeatable, objective assessment of conformity to
> specified requirements.
> --lynne
>
>
> At 06:23 PM 10/27/2003, Alex Rousskov wrote:
>
>
> >I owe Patrick Curran a blob on "lack of testability definition" for
> >TestGL. Here it is:
> >
> >         QA Framework in general and this document in particular
> >         rely on an an undefined term "testable" when defining and
> >         using key concepts such as test assertions. The lack of a
> >         definition for "testable" is not an oversight. The
> >         discussion and examples below explain why there is
> >         no good definition that is as general as QA Framework
> >         or TestGL scope.
> >
> >         Informally, "boolean condition X is testable" usually means
> >         "there exist a procedure to determine the truthfulness of X".
> >         A more specific (and less general) wording may require the
> >         "procedure" to be "finite" or even "affordable". The problem
> >         with this informal approach is that for most practical
> >         purposes related to computers, it is impossible to determine
> >         truthfulness by following a procedure. The best we can do is to
> >         attain a high level of confidence that X is true. Thus,
> >         the informal definition that most people would agree with
> >         may read "there exist a finite procedure to attain a high
> >         level of confidence that X is true". What level of
> >         confidence is "high enough" is, essentially, up to the
> >         tester to define for a given environment. The cost
> >         of the testing procedure and the cost of a false answer,
> >         among other factors, would affect that environment-specific
> >         definition.
> >
> >         To summarize, our inability to define "testable" lies
> >         in the probabilistic nature of computer-related tests
> >         and the fact that no general definition of "good
> >         probability" or "high enough confidence" can exist.
> >         The nature of the problem makes such definition
> >         very subjective and/or domain-specific.
> >
> >         For example, a "software system MUST accept invalid
> >         input" behavioral requirement can be tested using a black-box
> >         technique with all sorts of invalid inputs and under
> >         different conditions. However, there is no guarantee
> >         that the invalid input that crashes the system was
> >         tried. In most interesting cases the number of
> >         invalid inputs and/or conditions is infinite and not
> >         even countable. Whether, say, 99% confidence level would
> >         be acceptable for concluding conformance in this case
> >         would depend on things like software domain (isolated home PC
> >         versus nuclear plant). Not to mention that there are
> >         many ways to calculate that confidence level.
> >
> >         A less obvious example is a "document syntax conforms to
> >         specification X" formatting requirement. Even if
> >         specification X is written in a formal language, verifying
> >         conformance of large documents would be impractical without
> >         the use of automated validators. Thus, one would essentially
> >         reformulate the above requirement into something like
> >         "a correct document validator states that the
> >         document syntax conforms to specification X". The first
> >         sub-requirement ("the validator is correct") is equivalent
> >         to the behavioral example discussed above.
> >
> >
> >Please feel free to edit the above text or throw it away, of course.
> >
> >This submission is meant to complete AI-20031022-7, which is
> >mis-stated as "come up with the definition for the term testable" at
> >http://lists.w3.org/Archives/Public/www-qa-wg/2003Oct/0040.html
> >AI-20031022-7 is due October 29.
> >
> >Thank you,
> >
> >Alex.
>
>

Received on Friday, 31 October 2003 11:14:54 UTC