Re: Glossary: "Atomic Test"

David,

	I agree that single-minded test cases are needed and usually
preferred to the catch-all cases. When designing a test suite, I want
to be able provide a user with the exact failure location rather than
a generic "your product is not compliant" conclusion.

	From your example below and my experience, it looks like the
atomicity is based on the _intent_ of the test case (verifying a
single rule) and not on the coverage of the test case (exercising
several rules). Perhaps the definitions should be fixed with that idea
in mind.

Thanks,

Alex.


On Fri, 24 May 2002, David Marston/Cambridge/IBM wrote:

> Alex rousskov writes:
> >Atomic Test
> >1. Do we need this term?
> 
> >2. The definition says "maps back to exactly one [test] assertion".
> >Test assertion is defined as a "set of [...] rules that are known to
> >be true by definition in the spec". Since two sets of rules are still
> >a single set of rules, it is not clear what an "exactly one set of
> >rules" is. The whole standard is a single test assertion, by
> >definition.
> 
> >The definition further says "this is in contrast to some test cases
> >that may test a combination of rules". However, again, a combination
> >(set) of rules is a (single?) assertion, by definition.
> 
> I think we have a better grasp of what an atomic test *case* is than
> we do of an atomic test *assertion*, so I would argue that we should
> use the clear concept to push back on our notion of "test assertion"
> and ensure that the two mesh well.
> 
> An atomic test should test one path through the code, which typically
> means that a combination of conditions must be in effect. If you can
> distinguish "background" or "environmental" conditions from those
> conditions that are part of the test case, you have the potential to
> identify "molecular" tests as well as atomic ones.
> 
> A clear example is in the xsl:number facility of XSLT. See
> http://www.w3.org/TR/xslt#number
> and notice how hard it is to isolate the test assertions.
> When you allow xsl:number to calculate numbers based on the source
> document, the "level", "count", and "from" parameters all play a
> role in the calculation, whether specified or defaulted. Even if you
> think that "level" (a choice among three keywords) is part of the
> test background, all numbering uses the current node name (and other
> properties) along with the "count" and "from" patterns in an algorithm.
> The test assertions, and therefore the test cases, need to address
> several parameters of xsl:number at once. Whether the assertions are
> laid out atomically or not, the single-minded test case tries to
> number one kind of node under one set of level/count/from conditions
> and produce the asserted numbers. The benefit of these single-minded
> test cases (whether "atomic" or "molecular") is about isolating what
> is failing. For example, an XSLT processor might fail 8 xsl:number
> tests, and those tests may be exactly the set where "level" is "any"
> and "from" is a union of element names. There might be 8 tests because
> of different possibilities for "count" or the source data.
> 
> If we only care about pass/fail evaluation of the whole test subject,
> we might still want to retain "atomic test" in our glossary just to
> use it in conjunction with "test assertion", which has been a
> problematic term.
> .................David Marston
> 
> 
>

Received on Friday, 24 May 2002 11:18:55 UTC