Adapting library test suite for more general use?

Hi,

Last week during the BPWG call, I took an action item (ACTION-609) to
look at adapting the checker library test suite into a test suite that
would fit our CR exit criteria.

As a reminder, the CR exit criteria reads on this topic:
        existence of a test suite to verify the correct operation of
        mobileOK checkers.
        http://www.w3.org/TR/2007/CR-mobileOK-basic10-tests-20071130/#status
        
There are a few things that needs to be changed in the current library
test suite to make it fitting:
 * currently, our tests are unit tests - i.e. they'll only give the
results you get for a single module, rather than the results you would
get for a full checker; it's probably easy enough to provide both full
output and unit output, so I don't think this is much work
 * the results files currently include the specific textual messages
sent by our checker; given that these aren't required, we need results
file that don't include those - I'm thinking to have in the results file
only the tests-ids as identified in the spec, and their results
(pass/warning/fail)
 * running the test suite is currently very much bound to running the
Java servlet provided with the library unit-test framework; we may want
to provide the tests independently, but I don't consider this as
critical
 * the most important point is that we need to check that every single
test of the spec has a matching test case in our test suite; this in
turns implies that the ids we have in the test results should match the
ones defined in the spec (which wasn't the case for all of them last
time I checked)

So, I plan to work on implementing these changes in the next few weeks;
I'm assuming that most of the changes will be kept external to the
library, with e.g. a script that can build the official BPWG test suite
based on the checker test suite.

Feedback and suggestions are welcome,

Dom

Received on Wednesday, 12 December 2007 13:26:58 UTC