Re: lack of testability definition

At 11:29 AM 2003-11-04, Alex Rousskov wrote:
>On Tue, 4 Nov 2003, Al Gilman wrote:
>
> > <draft
> > class="strawman glossaryEntry notDefinition">
> >
> > A <term class="general">test</term> is a repeatable procedure for 
> confirming
> > or denying the truth value of a proposition.
> >
> > A proposition is <term class="general">testable</term> if there is such a
> > procedure that leaves little doubt as to the truth value of the proposition
> > after the procedure has been applied.
> >
> > </draft>
>
>I think the version that started this threat is a bit more accurate
>and positive (which is good for human perception). Rephrasing using
>your style:
>
>         A proposition is testable if there is such a procedure that
>         assesses the truth value of a proposition with a high enough
>         confidence level. Whether the confidence level is measurable
>         and what confidence level is high enough depends on the
>         proposition and its framework.

That is interesting.  We are very close, but your restatement discloses
a difference.  I don't feel that 'enough' belongs in a definition for our
use of this term.  Because 'enough' is not, in general, defined.

On the one hand it is true that most readers would interpret 'testable' to
indicate that the results of a test were dispositive, or that lay
interpreters of the test would interpret it as dispositive.  But we as
quality engineers are obliged as a professional duty to disrupt this pat
answer with the fact that this is not the full and final answer.

Our framework should not be predicated on an approximation that most people
would accept but which leads to wrong answers in engineering terms.

It is the existence of a way to bring material aid to resolving the
question that is the boolean I would call 'testable.'

The proposition can be testable even when you are left taking a calculated
risk after applying the test.  All that is required is that the test
introduces significant amounts of information into the calculation of your
prospects.

In particular, I would say a proposition is "marginally testable" when the
cost of making the test is on a par with the value of the information that
it returns.

The evidence and its statistical characterization roll forward into trust
calculations downstream.  Only sometimes is it appropriate for there to be a
boolean generated representing the results of a test case.  In the general
case.

So a framework, an application of these principles, MAY define 'enough' but
need not do so.

The most glaring example of the non-boolean nature of test results comes
from medical ethics, where the experimentation with a new clinical test
may be terminated before the originally planned length of the trial because
the evidence gathered to date demonstrates with sufficient confidence that
deployment of the test will save lives.  Even though it is at the same time
demonstrated that sometimes it will return erroneous results.

But where the rubber hits the road for the Web: assuring the interoperation
of one human user's understanding of a ballot or business transaction with
another human's understanding, it is particularly damaging to think that we
can or should reduce our systems of practice to a long list of checkpoints
with a Boolean 'and' function as the rollup operator.  This leaves the baby
out with the bathwater.  The value added from testing is particulary high
with respect to the squishy points.

Al

>Alex.

Received on Tuesday, 4 November 2003 13:20:54 UTC