Re: testability definition

On Fri, 6 Sep 2002, Lofton Henderson wrote:

> > >       "A specification is testable if there exists some finite
> > >       cost-effective process with which a person or software can check
> > >       that the requirement has been met."
>
> I think Alex is right, that we ought to have a look at the language
> here.  We are all aware that you can never prove behavioral
> specifications correct (but you can prove them incorrect).  Most of
> us at the same time believe in the value of applying falsification
> testing to implementations of behavioral requirements.
>
> As we have defined testability, it does not encompass the scope of
> things we would like to address in SpecGL, and not what we have in
> mind when we use a phrase like "better testability".  So we should
> adjust SpecGL's language so there is no potential inference that it
> -- the suitability of behavioral specifications for testing -- is
> out of scope.


How about this, to start with:

	"A requirement is testable if a finite cost-effective process
	 can detect any known a priory violation of the requirement.
	 A specification is testable if all its requirements are
	 testable."

The above needs polishing, but the idea is to base the definition on
detection of violations rather than detection of compliance. That is,
if Foo has no known violations it is deemed compliant.

The "known a priory" part is necessary to avoid implication that any
and all violations must be detectable. We can only detect violations
we test for (and hence know about).


As far as I can tell, the new definition is equivalent to the original
one when it comes to non-behavioral specs. In fact, even when
verifying conformance of documents with validators we implicitly use
the same trick -- if validator has a bug and does not check for
something, we may consider a invalid document valid. The only reason
we need a new definition is the difference between
	"there exists" (meaning "there could exist", of course)
	for non-behavioral specs
and
	"there does not exist" (meaning "there could never exist")
	for behavioral specs

$0.02,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Friday, 6 September 2002 18:27:01 UTC