Re: Glossary: "Atomic Test"

Yes, I agree -- it is true that in atomic test 
cases you are typcially testing more than 
one assertion -- it's intent is to isolate 
failures based on a single rule.

--Mary

----- Original Message ----- 
From: "Alex Rousskov" <rousskov@measurement-factory.com>
To: "David Marston/Cambridge/IBM" <david_marston@us.ibm.com>
Cc: <www-qa@w3.org>
Sent: Friday, May 24, 2002 11:18 AM
Subject: Re: Glossary: "Atomic Test"


> David,
> 
> I agree that single-minded test cases are needed and usually
> preferred to the catch-all cases. When designing a test suite, I want
> to be able provide a user with the exact failure location rather than
> a generic "your product is not compliant" conclusion.
> 
> From your example below and my experience, it looks like the
> atomicity is based on the _intent_ of the test case (verifying a
> single rule) and not on the coverage of the test case (exercising
> several rules). Perhaps the definitions should be fixed with that idea
> in mind.
> 
> Thanks,
> 
> Alex.
> 
> 
> On Fri, 24 May 2002, David Marston/Cambridge/IBM wrote:
> 
> > Alex rousskov writes:
> > >Atomic Test
> > >1. Do we need this term?
> > 
> > >2. The definition says "maps back to exactly one [test] assertion".
> > >Test assertion is defined as a "set of [...] rules that are known to
> > >be true by definition in the spec". Since two sets of rules are still
> > >a single set of rules, it is not clear what an "exactly one set of
> > >rules" is. The whole standard is a single test assertion, by
> > >definition.
> > 
> > >The definition further says "this is in contrast to some test cases
> > >that may test a combination of rules". However, again, a combination
> > >(set) of rules is a (single?) assertion, by definition.
> > 
> > I think we have a better grasp of what an atomic test *case* is than
> > we do of an atomic test *assertion*, so I would argue that we should
> > use the clear concept to push back on our notion of "test assertion"
> > and ensure that the two mesh well.
> > 
> > An atomic test should test one path through the code, which typically
> > means that a combination of conditions must be in effect. If you can
> > distinguish "background" or "environmental" conditions from those
> > conditions that are part of the test case, you have the potential to
> > identify "molecular" tests as well as atomic ones.
> > 
> > A clear example is in the xsl:number facility of XSLT. See
> > http://www.w3.org/TR/xslt#number
> > and notice how hard it is to isolate the test assertions.
> > When you allow xsl:number to calculate numbers based on the source
> > document, the "level", "count", and "from" parameters all play a
> > role in the calculation, whether specified or defaulted. Even if you
> > think that "level" (a choice among three keywords) is part of the
> > test background, all numbering uses the current node name (and other
> > properties) along with the "count" and "from" patterns in an algorithm.
> > The test assertions, and therefore the test cases, need to address
> > several parameters of xsl:number at once. Whether the assertions are
> > laid out atomically or not, the single-minded test case tries to
> > number one kind of node under one set of level/count/from conditions
> > and produce the asserted numbers. The benefit of these single-minded
> > test cases (whether "atomic" or "molecular") is about isolating what
> > is failing. For example, an XSLT processor might fail 8 xsl:number
> > tests, and those tests may be exactly the set where "level" is "any"
> > and "from" is a union of element names. There might be 8 tests because
> > of different possibilities for "count" or the source data.
> > 
> > If we only care about pass/fail evaluation of the whole test subject,
> > we might still want to retain "atomic test" in our glossary just to
> > use it in conjunction with "test assertion", which has been a
> > problematic term.
> > .................David Marston
> > 
> > 
> > 
> 
>

Received on Friday, 24 May 2002 11:28:17 UTC