Re: Test harness requirements

On 05/04/2010 04:09 PM, Geoffrey Sneddon wrote:

> As was said on the previous telecon, neither myself nor James have any
> objection to approving the test as is; however, I still think we do need
> to design the harness to deal with everything it needs to do initially,
> as a number of test harness really suffer from having things like
> support for async tests hacked on to them at a later date. This will
> mean moving tests already approved over to a new harness at a later
> date, and that we probably don't want to approve large test suites until
> we know what the API will be.


So on the subject of test harnesses, I sketched out an idea-as code [1]. 
It is not supposed to be complete; in particular it should have more 
assert_* functions and better output (including the source of the test 
steps, for example). There is also an example test document showing 
roughly how the harness works. Hopefully that is self-explanatory but, 
in case it is not:

The harness exposes a small number of global functions, the most 
important being "test" and "test_async". The test function creates and 
runs a test synchronously. It takes a function that contains the test 
code, and a name for the test. test_async allows for asynchronous tests, 
it takes the name of a test and returns a test object. The "step" method 
on that test object can be called at any later time taking a function 
that runs further assertions for this test. Once the test is complete 
(no more steps will be run), the "done" method on the test object must 
be called. If the done method is not called a certain time after the 
test starts then it times out. All calls to test and test_async must 
occur before the load event fires.

Assertions are provided by assert_* functions. At the moment 
assert_true, assert_equals, assert_object_equals and assert_exists. More 
are needed to cover common things like checking that a function throws, 
and checking various IDL properties (e.g. assert_readonly).

To enable integration with browser regression test systems, there are a 
couple of functions for registering results handlers; 
add_results_callback and add_completion_callback. One is called once per 
result, the other is called when all results are available.

Hopefully that is enough information to start some discussion about 
strengths and weaknesses so we can iterate in the direction of a good 
design.

[1] http://hg.hoppipolla.co.uk/hgwebdir.cgi/domharness/

Received on Tuesday, 18 May 2010 09:28:12 UTC