Re: lack of testability definition

At 03:38 PM 2003-11-03, Karl Dubost wrote:

>Alex,
>
>thank you for this definition.
>
>I think that we are reaching the disagreement I was feeling at the last 
>F2F meeting about testability. It might be that my understanding of 
>testability is based on another assumption.
>
>For me, I would say that everything is testable without ambiguity. I would 
>say that every tests have a 3 possible states.
>
>         Yes, No and Not Applicable.

This theory is contrary to the preponderance of received wisdom in Quality,
which is based in statistical models.

W3C seeks to advance interoperability in actual practice.  Interoperability 
in the actual practice is statistical; not 100%.

Software today is mostly *not* programmed in provable media.  So the
evidence developed by software testing is inappropriate to view as boolean
absolute truth values.  Statistical assesment of the strength of the
evidence is appropriate; except in rare cases absolute certainty is not
achievable from tests that we are going to get people to apply.  So don't go
there.

You lose, on the 'everything is testable' claim but you win in terms of
"now we can say why there is no boolean-certain definition of 'testable'
in this framework.

Start over with statistics, and you will be using a meaning for 'testable'
that both the software industry and the other stakeholders in the
interoperability of their software can subscribe to.

<draft
class="strawman glossaryEntry notDefinition">

A <term class="general">test</term> is a repeatable procedure for confirming
or denying the truth value of a proposition.

A proposition is <term class="general">testable</term> if there is such a
procedure that leaves little doubt as to the truth value of the proposition
after the procedure has been applied.

</draft>

The above is to be considered a re-telling of colloquial usage, not a
definition in the sense of an allusion to a notion that can be evaluated as
a Boolean predicate.

The magnitude of "little doubt" is not to be frozen once for all domains
where the QA documents are to be applied.  The definition in the WCAG
Techniques Requirements that Bjorn cited is overkill in that regard.  The
qualifications for a procedure to be repeatable enough to merit recognition
as a test will likewise vary from domain to domain of the actual end
application.

So this is the sense in which there should *not* be a 'definition' of
'testable' in these documents.  The above common usage, restated for the
assistance of the readers, should be all the 'definition' that there is.
Applications of the QA framework documents may refine this by their own Term
of Art where it is constructive, but should think three times before they
attempt to do so.

Al

>You have to be sure when you design the test that "not applicable" is not 
>part of the nature of the test itself, but one of the possible choices.
>
>
>Le Lundi, 27 octo 2003, à 18:23 America/Montreal, Alex Rousskov a écrit :
>>         Informally, "boolean condition X is testable" usually means
>>         "there exist a procedure to determine the truthfulness of X".
>>         A more specific (and less general) wording may require the
>>         "procedure" to be "finite" or even "affordable". The problem
>>         with this informal approach is that for most practical
>>         purposes related to computers, it is impossible to determine
>>         truthfulness by following a procedure. The best we can do is to
>>         attain a high level of confidence that X is true. Thus,
>
>For me, the assertion, you made in this paragraph is false. In the sense 
>that the computers are not less testable than physics phenomenons. (Here, 
>I have chosen Physics on purpose, that's my background and you have 
>uncertainty laws like the position of an electron around an hydrogen atom).
>
>In the assertion made here, you just show that you test the wrong thing 
>and it's usually an error that many people do. :)
>
>In Physics, you can't know the position of an electron around a proton in 
>an hydrogen atom. But quantum physics gives you a framework which gives 
>you the probability of finding the electron at a certain position.
>
>*******
>You can't test the exact position, so it's not a test.
>But you can test if the series of measurements respect the physics law.
>*****
>
>The nature of testability is not intrinsec to the topic itself (computers, 
>physics, etc) but to the way you define your tests.
>
>So for me, there's no such thing like lack of testability. Everything is 
>testable, it only depends on the way you define your tests. If the results 
>of the test is unpredictable, it's because your test assertion has an issue.
>
>Let's come back to a W3C specification.
>
>         alt="text"
>
>         the value of the alt attribute specifies a text of replacement 
> when the image is not displayed.
>
>         If your test criteria, is that, the content of the alt attribute 
> must be identically the same. The test is not good. It doesn't mean that 
> the alt attribute is not testable, but that your test criteria is not 
> well defined.
>
>
>I'm not argueing about the definition, but more about the door we are 
>opening. If we define things like not testable, it means that people will 
>come with features in specification which are not testable and declared as 
>it, which is dangerous.
>
>
>
>--
>Karl Dubost - http://www.w3.org/People/karl/
>W3C Conformance Manager
>*** Be Strict To Be Cool ***

Received on Tuesday, 4 November 2003 10:55:48 UTC