Re: lack of testability definition

On Mon, 3 Nov 2003, Karl Dubost wrote:

> Physics laws are models... there's nothing true per se. As soon you
> have decided of a framework, your tests have to be verifiable in
> this model, when your model is not anymore good because too many
> cases are out of the model, you have to change the model.

OK. I may have a better understanding of your position now.
Essentially, you add another variable to the picture. You call that
variable a "framework". In math, that is usually called a set of
axioms. Axioms are not true or false. They just are. Like believes.
One can change the axiom and such a change would propagate to all
derived laws and experiments.

If that is the essence of what you are saying, then I agree in
principle. The (framework + specification) model is correct. However,
everything QA WG have been writing about so far uses a simpler model
that assumes that specification is the only "input" and incorporates
any framework assumptions. In other words, your current documents
assume that framework is built into the spec (which is only partially
true).

> > Consider this instead: Is the following statement from a spec
> > testable?
> >
> > 	"When given input X, the agent MUST always respond with Y?"
> >
> > If yes, please provide a procedure to establish compliance of a
> > given agent implementation.
>
> yes if the result Y is unpredictable, you just proved the test
> assertion is not true ;) and so you have to reformulate it. If you
> can't it means that the framework is not good.

Output predictability depends on the testing framework. For example,
if you test only on MS Windows platform and only on Saturdays, you
always get Y. If you test on Linux on the first Monday of the month,
you always get Z. You do not know either of that a priory, of course.
You try to test under as many conditions as possible in hope to gain
enough confidence that the agent always responds with Y.

Using your framework model, you can say that a framework that does not
include Monday tests on Linux is no good if the agent is expected to
run on Linux on Mondays. I say that, while this may be true, it does
not help anybody in practice because they cannot try all possible
combinations. In other words, they cannot find a good framework and be
able to verify that it is a good framework. That is why my suggestion
is to (a) acknowledge the problem that the exact framework is
undefined by most specifications and (b) encourage QA folks to think
carefully about test coverage given the fuzziness of the framework.

Alex.

Received on Tuesday, 4 November 2003 11:02:48 UTC