Re: lack of testability definition

On Mon, 3 Nov 2003, Karl Dubost wrote:

> For me, I would say that everything is testable without ambiguity.

Is your own statement that "everything is testable without ambiguity"
is testable without ambiguity? :-)

> > 	Informally, "boolean condition X is testable" usually means
> > 	"there exist a procedure to determine the truthfulness of X".
> > 	A more specific (and less general) wording may require the
> > 	"procedure" to be "finite" or even "affordable". The problem
> > 	with this informal approach is that for most practical
> > 	purposes related to computers, it is impossible to determine
> > 	truthfulness by following a procedure. The best we can do is to
> > 	attain a high level of confidence that X is true. Thus,
>
> For me, the assertion, you made in this paragraph is false. In the
> sense that the computers are not less testable than physics
> phenomenons. (Here, I have chosen Physics on purpose, that's my
> background and you have uncertainty laws like the position of an
> electron around an hydrogen atom).
>
> In the assertion made here, you just show that you test the wrong thing
> and it's usually an error that many people do. :)
>
> In Physics, you can't know the position of an electron around a
> proton in an hydrogen atom. But quantum physics gives you a
> framework which gives you the probability of finding the electron at
> a certain position.
>
> *******
> You can't test the exact position, so it's not a test.
> But you can test if the series of measurements respect the physics law.
> *****

Only to a certain degree of confidence. As you know from the history
of physics, virtually every theory of "how the world works" was
supported by experimental data. Before "dark ages", experiments were
pretty bad and often confirmed wrong theories! Then QA got better
and/or more experiment data was available. What does it tell us? It
tells us that you cannot prove something 100% by testing.

I am not a physics guru, but my understanding is that the base
principles of quantum mechanics are still in dispute and will be so at
least until a Grand Unified Theory is completed.

In other words, the assertion you call the "physics law" can be
subjected to a series of test cases, but no series of test cases can
prove 100% that the law is "true". We always find flaws when we dig
deeper and as our experimental ability improves.

> The nature of testability is not intrinsec to the topic itself
> (computers, physics, etc) but to the way you define your tests.
>
> So for me, there's no such thing like lack of testability.
> Everything is testable, it only depends on the way you define your
> tests. If the results of the test is unpredictable, it's because
> your test assertion has an issue.

Fine, but this means that you just shifted the problem to the
definition of issue-less test assertion. In other words, if we accept
your plan, then we can agree that it easy to define what is testable,
but we have to agree that it is impossible to define what test
assertions are good enough to prove that the requirements are met.

Recall that we want to test conformance to spec requirements;
everything else (assertions, test cases, etc.) are just means of
accomplishing our goal.

> Let's come back to a W3C specification.
>
> 	alt="text"
>
> 	the value of the alt attribute specifies a text of replacement when
> the image is not displayed.
>
> 	If your test criteria, is that, the content of the alt attribute must
> be identically the same. The test is not good. It doesn't mean that the
> alt attribute is not testable, but that your test criteria is not well
> defined.

I agree. This example does not illustrate the problem though.
Consider this instead: Is the following statement from a spec
testable?

	"When given input X, the agent MUST always respond with Y?"

If yes, please provide a procedure to establish compliance of a given
agent implementation.

> I'm not argueing about the definition, but more about the door we
> are opening. If we define things like not testable, it means that
> people will come with features in specification which are not
> testable and declared as it, which is dangerous.

I do not suggest that we encourage people to have untestable
statements, but I believe that specifications will always have
untestable (in my opinion) statements. I suggest that we either:

	1) Define what is testable, then say that every spec
	   requirement SHOULD be testable using our definition
or
	2) Define that testability is tester-specific, then
	   say that every spec SHOULD be testable using
	   authors definition of testability.

Alex.

Received on Monday, 3 November 2003 19:38:54 UTC