RE: testability definition (was: Re: FYI : a new draft of "QA Framework - Specifications Guidelines" published)

Why not remove "finite cost-effective" and leave:
> >       A specification is testable if there exists some 
> >       process with which a person or software can check
> >       that the requirement has been met."

IMO all we want to say is that a (behavioral) specification is testable if one can deduce a (unambiguous) behavior from it for any input allowed by the specification's grammar.

Then the proposed criteria is:
The spec is testable as long as you can build the process (not necessarily finite) which: 
-  enumerates all possible inputs (true in case of both HTTP and SOAP)
-  (unambiguously) deduces the behavior from the spec for each generated input 

One can check this criteria for any behavior spec I know of.


-----Original Message-----
From: Lofton Henderson [mailto:lofton@rockynet.com] 
Sent: Friday, September 06, 2002 3:03 PM
To: Alex Rousskov; Dominique Hazaël-Massieux
Cc: www-qa@w3.org
Subject: testability definition (was: Re: FYI : a new draft of "QA Framework - Specifications Guidelines" published)



> >       "A specification is testable if there exists some finite
> >       cost-effective process with which a person or software can check
> >       that the requirement has been met."

At 10:49 AM 9/5/02 -0600, Alex Rousskov wrote:

>On 3 Sep 2002, Dominique [ISO-8859-1] Hazaël-Massieux wrote:
>
> > > Virtually all specifications that define behavior based on inputs are
> > > not testable according to the above definition. There is no finite
> > > cost-effective process to enumerate all valid inputs. Yet such an
> > > enumeration would be required to check that implementation meets the
> > > requirement. The guidelines should either limit their scope to
> > > non-behavioral specifications or relax the definition of testability.
> >
> > I'm not sure about this: why should we relax this definition? We're
> > just asserting that a behavioral specification is not testable in
> > our definition. We're not saying that it's bad, nor that our
> > guidelines apply or not to it...
>
>You are not saying, you may be implying. If that's what you want to
>say then:
>
>         (a) I would suggest that the statement is made explicit:
>             "behavioral specifications (e.g., SOAP, SMIL?, HTTP) are
>             defined to be not testable"
>
>         (b) I am surprised WG wants to spend so much resources on
>             something that immediately alienates future (and
>             current) behavioral specifications
>
>         (c) I would recommend relaxing the definition so that
>             the resources spent on producing SpecGL are better
>             utilized (as they would apply to more specs). Or does
>             QAWG plan to write BehaveGL next?

I think Alex is right, that we ought to have a look at the language 
here.  We are all aware that you can never prove behavioral specifications 
correct (but you can prove them incorrect).  Most of us at the same time 
believe in the value of applying falsification testing to implementations 
of behavioral requirements.

As we have defined testability, it does not encompass the scope of things 
we would like to address in SpecGL, and not what we have in mind when we 
use a phrase like "better testability".  So we should adjust SpecGL's 
language so there is no potential inference that it  -- the suitability of 
behavioral specifications for testing -- is out of scope.

-Lofton.

Received on Friday, 6 September 2002 18:36:52 UTC