TEST: scope

I wished to write down my concerns about the scoping of the TEST work.


Test suites have many different purposes:
- checking correctness
   - at particular points (issue driven)
   - generally (conformance testing)
- exercising difficult problems
- performance testing
- scale testing

Test suites have at least two different audiences:
- systems and their developers
- other humans

The latter audience prefer small tests that can be easily understood.
This audience may be reading the tests in order to better understand some text.

=========================


My view is that we need to be clear as to what we are trying to achieve.

I suggest that we should generate tests that illustrate our issue resolutions.
I suggest that we should keep all our tests as small as possible.

I suggest that we should not aim at a conformance test suite.
I suggest that performance and scalability tests (and the like) are out of
scope.

I think some of Ian's tests are
- exercising difficult problems
and
- as small as they can be to illustrate the issue.
(Modulo the OILedit => DAML+OIL conversion which appears a bit verbose!)

As such I think I support them ...

===

We also need to get process in place to generate and agree test cases.
Each test case is quite expensive and in my view we should not generate too
many. (Not that there is any danger at the moment!)

Smaller tests are much cheaper than bigger tests because they are much more
likely to be right first time.

Jeremy

Received on Wednesday, 26 June 2002 05:49:16 UTC