lack of testability definition

I owe Patrick Curran a blob on "lack of testability definition" for
TestGL. Here it is:

	QA Framework in general and this document in particular
	rely on an an undefined term "testable" when defining and
	using key concepts such as test assertions. The lack of a
	definition for "testable" is not an oversight. The
	discussion and examples below explain why there is
	no good definition that is as general as QA Framework
	or TestGL scope.

	Informally, "boolean condition X is testable" usually means
	"there exist a procedure to determine the truthfulness of X".
	A more specific (and less general) wording may require the
	"procedure" to be "finite" or even "affordable". The problem
	with this informal approach is that for most practical
	purposes related to computers, it is impossible to determine
	truthfulness by following a procedure. The best we can do is to
	attain a high level of confidence that X is true. Thus,
	the informal definition that most people would agree with
	may read "there exist a finite procedure to attain a high
	level of confidence that X is true". What level of
	confidence is "high enough" is, essentially, up to the
	tester to define for a given environment. The cost
	of the testing procedure and the cost of a false answer,
	among other factors, would affect that environment-specific
	definition.

	To summarize, our inability to define "testable" lies
	in the probabilistic nature of computer-related tests
	and the fact that no general definition of "good
	probability" or "high enough confidence" can exist.
	The nature of the problem makes such definition
	very subjective and/or domain-specific.

	For example, a "software system MUST accept invalid
	input" behavioral requirement can be tested using a black-box
	technique with all sorts of invalid inputs and under
	different conditions. However, there is no guarantee
	that the invalid input that crashes the system was
	tried. In most interesting cases the number of
	invalid inputs and/or conditions is infinite and not
	even countable. Whether, say, 99% confidence level would
	be acceptable for concluding conformance in this case
	would depend on things like software domain (isolated home PC
	versus nuclear plant). Not to mention that there are
	many ways to calculate that confidence level.

	A less obvious example is a "document syntax conforms to
	specification X" formatting requirement. Even if
	specification X is written in a formal language, verifying
	conformance of large documents would be impractical without
	the use of automated validators. Thus, one would essentially
	reformulate the above requirement into something like
	"a correct document validator states that the
	document syntax conforms to specification X". The first
	sub-requirement ("the validator is correct") is equivalent
	to the behavioral example discussed above.


Please feel free to edit the above text or throw it away, of course.

This submission is meant to complete AI-20031022-7, which is
mis-stated as "come up with the definition for the term testable" at
http://lists.w3.org/Archives/Public/www-qa-wg/2003Oct/0040.html
AI-20031022-7 is due October 29.

Thank you,

Alex.

Received on Monday, 27 October 2003 18:23:51 UTC