Re: testability definition (was: Re: FYI : a new draft of "QA Framework - Specifications Guidelines" published)

At 04:03 PM 9/6/2002 -0600, Lofton Henderson wrote:


>> >       "A specification is testable if there exists some finite
>> >       cost-effective process with which a person or software can check
>> >       that the requirement has been met."
>
>At 10:49 AM 9/5/02 -0600, Alex Rousskov wrote:
>
>>
>> > > Virtually all specifications that define behavior based on inputs are
>> > > not testable according to the above definition. There is no finite
>> > > cost-effective process to enumerate all valid inputs. Yet such an
>> > > enumeration would be required to check that implementation meets the
>> > > requirement. The guidelines should either limit their scope to
>> > > non-behavioral specifications or relax the definition of testability.
>
>I think Alex is right, that we ought to have a look at the language 
>here.  We are all aware that you can never prove behavioral specifications 
>correct (but you can prove them incorrect).  Most of us at the same time 
>believe in the value of applying falsification testing to implementations 
>of behavioral requirements.



Let me try to bring al little perspective back to this discussion.   The 
original definition (which I think is perfectly acceptable) says absolutely 
nothing, and implies absolutely nothing,  about "proving an implementation 
correct".  It's addressing the testability of the spec - not the provable 
correctness of the implementation.

Let's break it down.  It says:

1)  A spec  is testable

That's what we're defining

2) if there exists a finite, cost-effective process

Test suites we're familiar with are finite, (cost-effective is debatable), 
processes

3)  which a person or software can check  that the requirement has been met

Software (perhaps combined with human judgement) does "check" that the 
requirement has been met.  It may not "prove" that the requirement has been 
met, but it certainly checks.  In fact, if done correctly, it "checks" with 
a high degree of probability that the checking is correct.  Again, nowhere 
does this definition say "There is a finite  cost-effective process to 
enumerate all valid inputs." like Alex seems to think it says.

I believe this definition was actually thought out carefully.

It's correct and there is no reason to change it.  The only change I would 
contemplate is removal of the "cost effective" phrase.  It is too subjective.

Mark