Re: Should Test Assertions be required

On Fri, 6 Jun 2003, Lynne Rosenthal wrote:

> Should specification developers be required to include Test
> Assertions (TA)  in their specifications?

No. The goal of a specification is to document and explain format
and/or behavior, not to test a format or behavior of an
implementation.

> some Pros:
> 1. ensures testable specifications

It does not. It decreases the number of untestable requirements due to
poor wording and such. That side-effect can be achieved directly by
other means (see last paragraph of this e-mail).

> 2. consensus in what is required

Not at all. Consensus on how each testable requirement can be tested
at a minimal level. This usually boils down to a simple example
illustrating a requirement.

> 3. facilitates test generation

True. However, it may facilitate generation of both inferior
implementations and inferior tests because:

	- Spec authors usually do not care about test cases
	  much so they are not going to spend much time developing
	  decent TAs and they will have just the minimum set of
	  cases to cover all testable statements once.

	- Implementors with little resources may focus on satisfying
	  (coding for) documented TAs instead of satisfying the
	  specification itself. A very common pitfall, IMO.

	- Testers will have to implement documented TAs and may
	  stop there, instead of developing a more complete and
	  independent test suite.

By forcing spec authors to do testing we assign an important task to
the wrong people. Look, for example, at test cases for SOAP at
http://www.w3.org/TR/soap12-testcollection/ I am not a SOAP expert,
but these look a lot more like examples then quality test cases. A
decent test suite would do much better than that.

> IMO specifications should (must?) identify the testable statements
> or test requirements.

Specifications MUST define conformance criteria. Identifying testable
part of that criteria would be useful, but since "testable" is not
easy or even possible to define in general, we may not be able to
require such identification.

> Lets think about the objective - identifying the testable statements
> and facilitate the building of test suites - without going overboard
> and burdening the specification developer.  So, what would be
> approparite?

Authors MUST identify conformance-affecting statements (e.g., by using
MUST/SHOULD keywords). Authors MUST consider testability of each
identified conformance-affecting statement. The last requirement is
not testable.

HTH,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Friday, 6 June 2003 11:27:10 UTC