W3C home > Mailing lists > Public > www-qa@w3.org > September 2002

RE: testability definition

From: Kirill Gavrylyuk <kirillg@microsoft.com>
Date: Fri, 6 Sep 2002 15:54:50 -0700
Message-ID: <B3F0DACD72892E4DB7E8296C6C9FC2F6050753D9@red-msg-03.redmond.corp.microsoft.com>
To: "Alex Rousskov" <rousskov@measurement-factory.com>, "Lofton Henderson" <lofton@rockynet.com>
Cc: <www-qa@w3.org>

"A requirement is testable if a finite cost-effective process
	 can detect any known a priory violation of the requirement.
	 A specification is testable if all its requirements are
	 testable."

The problem is that it is not clear how do you deduce the requirements
from a spec. You need to be able to build a process that would deduce
all the requirements from the spec. 
There are buggy untestable specs that would satisfy the definition above
just because they do not specify the behavior for some inputs. 

Consider a testable behavioral spec as a many:1 map between an input
space and a behaviors space. 

The spec is in trouble if either >1 behaviors are specified for the same
input or none of the behaviors are specified for an input.

By enumerating the input space you can prove the testability.

Thoughts?


-----Original Message-----
From: Alex Rousskov [mailto:rousskov@measurement-factory.com] 
Sent: Friday, September 06, 2002 3:27 PM
To: Lofton Henderson
Cc: www-qa@w3.org
Subject: Re: testability definition


On Fri, 6 Sep 2002, Lofton Henderson wrote:

> > >       "A specification is testable if there exists some finite
> > >       cost-effective process with which a person or software can
check
> > >       that the requirement has been met."
>
> I think Alex is right, that we ought to have a look at the language
> here.  We are all aware that you can never prove behavioral
> specifications correct (but you can prove them incorrect).  Most of
> us at the same time believe in the value of applying falsification
> testing to implementations of behavioral requirements.
>
> As we have defined testability, it does not encompass the scope of
> things we would like to address in SpecGL, and not what we have in
> mind when we use a phrase like "better testability".  So we should
> adjust SpecGL's language so there is no potential inference that it
> -- the suitability of behavioral specifications for testing -- is
> out of scope.


How about this, to start with:

	"A requirement is testable if a finite cost-effective process
	 can detect any known a priory violation of the requirement.
	 A specification is testable if all its requirements are
	 testable."

The above needs polishing, but the idea is to base the definition on
detection of violations rather than detection of compliance. That is,
if Foo has no known violations it is deemed compliant.

The "known a priory" part is necessary to avoid implication that any
and all violations must be detectable. We can only detect violations
we test for (and hence know about).


As far as I can tell, the new definition is equivalent to the original
one when it comes to non-behavioral specs. In fact, even when
verifying conformance of documents with validators we implicitly use
the same trick -- if validator has a bug and does not check for
something, we may consider a invalid document valid. The only reason
we need a new definition is the difference between
	"there exists" (meaning "there could exist", of course)
	for non-behavioral specs
and
	"there does not exist" (meaning "there could never exist")
	for behavioral specs

$0.02,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance
Received on Friday, 6 September 2002 18:55:22 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:13:59 GMT