W3C home > Mailing lists > Public > www-qa@w3.org > September 2002

RE: testability definition

From: Alex Rousskov <rousskov@measurement-factory.com>
Date: Mon, 9 Sep 2002 10:33:29 -0600 (MDT)
To: Lynne Rosenthal <lynne.rosenthal@nist.gov>
cc: www-qa@w3.org
Message-ID: <Pine.BSF.4.44.0209090956310.443-100000@measurement-factory.com>

On Mon, 9 Sep 2002, Lynne Rosenthal wrote:

> I think the definition is fine as it is - after removing
> "cost-effective".

What is your definition/interpretation of "to check that requirement
has been met"?

> We may want to add, A testable specification uses concrete terms and
> measurable quantities, and avoid words such as "works well", "looks
> good" or "shall usually happen".

This is not related to the core discussion but please note that
"concrete" itself need to be defined for such an addition to have any
value. Also, "works well" and "looks good" is perfectly fine as long
as "well" and "good" are defined by the spec.

> Additionally any ambiguous requirement is not testable.

Again, "ambiguity" needs to be defined.

> This definition is consistent with an IEEE definition - which goes
> on to say: -- Statements such as "the program shall never enter an
> infinite loop" is non-testable, because the testing of this quality
> is theoretically impossible.  An example of a testable statement is:
> Output of the program shall be produced within 20 s of event x 60%
> of the time; and shall be produced within 30 s of event x 100% of
> the time.  This statement can be tested because it uses concrete
> terms and measurable quantities.

What about "proxy MUST forward URLs of any length"?
Concrete terms. URL length is measurable. Is this testable?

The answer depends on your definition of testable. "Yes" by my
definition. "No" by the current SpecGL definition in my
interpretation. "Yes" by the current SpecGL definition in Mark Skall's
interpretation.

> If a method cannot be devised to determine whether the software
> meets a particular requirement, then that requirement should be
> removed or revised."

It is interesting that IEEE does not say "CHECK that the software
meets a particular requirement". SpecGL's current definition, in Mark
Skall's interpretation is a lot broader that what IEEE seems to allow.


Again, my original concern is that the group accepts a too rigid
definition that excludes many specs. As a feedback, I am getting both
	"no, we do not exclude anything
	 because the definition must be interpreted differently"
AND
	"yes, we want to exclude most behavioral specs"

To me, this is a clear sign that even proponents of the current
definition disagree on its interpretation and even intent. Yet, they
all say that the current definition is "fine"! Weird. It is as if
people are not listening to each other while defending the current
status quo.

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance
Received on Monday, 9 September 2002 12:33:33 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:13:59 GMT