test cases -> running tests

Bjoern has done a pretty good job of laying out the basis of what we 
could run for an automated test suite of the validator, but we still 
need a mechanism to link or include the test cases themselves into this 
test system.

We have quite a lot of flexibility in this regard, given how we have 
not yet made any decision on how the test cases would be managed and 
stored.

A few possible solutions (some of them probably worth striking out 
immediatly):

1- test cases are just (self descripting) files in several directories, 
test program scans dir
         - too simple. no metadata at all. worse than what we have ATM.

2- "generate" code on demand based on repository of test cases and test 
code template
         - complicated, hard to maintain.

3- metadata written directly in [separate] perl code (use TestCases;)
         - not very easy to reuse metadata in other contexts

4- same as 3-, but metadata managed in config::general friendly text 
files
         - not very different from 3-, only easier to maintain

5- same as 4-, but metadata in some other language (XML, XHTML, other)
		- more flexible, can even imagine retrieving/merging test cases 
(meta)data from several sources

I am strongly leaning toward 5, but 4 could be a good tradeoff, given 
how we are/will be maintaining all our configuration with C::G 
maintaining the test cases list and metadata this way would reuse good 
chunks of code.

-- 
olivier

Received on Thursday, 21 October 2004 06:29:17 UTC