Re: some questions about version information in the test suite

>
> Since our requirement for getting to PR is to show that the
> changes between 1.0 and 1.1 are implementable and have been
> tested to show they have been successfully implemented, the
> class of tests for which results differ on the same input
> is crucially important to me.  Making those tests into pairs
> or larger sets of tests is a large step toward making it harder
> to get to PR.  Is there some compensating virtues that I am
> missing?
>

Conceptually I think we have a set of triples (I, C, O) where I is the 
input to the test, C is the conditions/configuration under which it 
should be run, and O is the expected output. Perhaps we are debating 
whether or not it is useful to group these by common I. I think the test 
driver is a little simpler if we don't (i.e. if we keep it flat) and 
reporting/analyzing results might also be a bit easier that way, while 
maintenance of the test suite is perhaps a little simpler if it is grouped.

What's a muddle, though, is if we do a bit of both.

The flat model would be along the lines

<test>
<input>...</input>
<conditions>...</conditions>
<result>...</result>
</test>

<test>
<input>...</input>
<conditions>...</conditions>
<result>...</result>
</test>

The grouped model would be more like

<test>
<input>...</input>
<outcome>
<conditions>...</conditions>
<result>...</result>
</outcome>
<outcome>
<conditions>...</conditions>
<result>...</result>
</outcome>
</test>

(Of course, if the condition is simple it can be made an attribute of 
<result> and outcome can then be collapsed; but I'd suggest avoiding 
that, because the conditions can become complex over time).

I've abstracted away from detailed questions about what <test> is: but 
in practice this is relevant because we have a testGroup containing a 
number of tests each with its own outcome, and this makes it a bit 
difficult to use the grouped structure if the same conditions apply to 
all tests within the group. I think it's this complexity that tends to 
lead me to the "flat" structure, where the conditions (including of 
course the version) are always written at the level of the <testGroup> 
element.

Michael Kay

Received on Friday, 18 June 2010 16:25:08 UTC