Glossary: "Atomic Test"

Alex rousskov writes:
>Atomic Test
>1. Do we need this term?

>2. The definition says "maps back to exactly one [test] assertion".
>Test assertion is defined as a "set of [...] rules that are known to
>be true by definition in the spec". Since two sets of rules are still
>a single set of rules, it is not clear what an "exactly one set of
>rules" is. The whole standard is a single test assertion, by
>definition.

>The definition further says "this is in contrast to some test cases
>that may test a combination of rules". However, again, a combination
>(set) of rules is a (single?) assertion, by definition.

I think we have a better grasp of what an atomic test *case* is than
we do of an atomic test *assertion*, so I would argue that we should
use the clear concept to push back on our notion of "test assertion"
and ensure that the two mesh well.

An atomic test should test one path through the code, which typically
means that a combination of conditions must be in effect. If you can
distinguish "background" or "environmental" conditions from those
conditions that are part of the test case, you have the potential to
identify "molecular" tests as well as atomic ones.

A clear example is in the xsl:number facility of XSLT. See
http://www.w3.org/TR/xslt#number
and notice how hard it is to isolate the test assertions.
When you allow xsl:number to calculate numbers based on the source
document, the "level", "count", and "from" parameters all play a
role in the calculation, whether specified or defaulted. Even if you
think that "level" (a choice among three keywords) is part of the
test background, all numbering uses the current node name (and other
properties) along with the "count" and "from" patterns in an algorithm.
The test assertions, and therefore the test cases, need to address
several parameters of xsl:number at once. Whether the assertions are
laid out atomically or not, the single-minded test case tries to
number one kind of node under one set of level/count/from conditions
and produce the asserted numbers. The benefit of these single-minded
test cases (whether "atomic" or "molecular") is about isolating what
is failing. For example, an XSLT processor might fail 8 xsl:number
tests, and those tests may be exactly the set where "level" is "any"
and "from" is a union of element names. There might be 8 tests because
of different possibilities for "count" or the source data.

If we only care about pass/fail evaluation of the whole test subject,
we might still want to retain "atomic test" in our glossary just to
use it in conjunction with "test assertion", which has been a
problematic term.
.................David Marston

Received on Friday, 24 May 2002 11:00:18 UTC