Test harness requirements

On 04/20/2010 04:33 PM, Kris Krueger wrote:
> Given the size of the HTML5 spec, I would hope that many people contribute tests to the suite.
> For the DOM tests that get submitted, I'd suggest the following as potential requirements.
> * Simple Run (Just Load The Page)
> * Pass/Fail Results are clear and consistent
> * All the core parts (not tests live in a single js file, e.g. asserts)
> * No complex design - hidden iframes, page load events, etc..

OK, so this is a harness that we want to be reused. In that case I guess 
it is worthwhile providing some detailed feedback and worthwhile for the 
group to spend some time iterating the harness design (this need not be 
a blocker to approving the existing testcase as long as we are prepared 
to modify the existing test to match any modifications to the harness).

Here is some initial thinking (not, I am not saying that the current 
harness does poorly at all these things, I am just trying to produce a 
reasonably complete set of good features as a starting point for further 

== Requirements / desirable features ==

* Tests run automatically

* Binary results; each test is either a pass or a fail (possibly with 
various subtypes of fail to distinguish e.g. timeouts from failed 

* Readable tests; assertions should be clearly visible in the source and 
should be self-documenting i.e. it should be clear from the harness API 
what type of assertion is being made

* Granular tests; design should encourage testing of one thing per test 
so that each failure can be associate with a single bug (so e.g. looping 
over many properties in a single test is bad because you can't tell 
which one failed)

* Independent tests; failure of a single test should not change the 
results of other tests

* Good error messages; the framework should enable the production of 
good error messages that make it clear what failed and help debugging

* Strict checking; equality comparisons should not typically do implicit 
type conversion

* Support for deep object equality testing; ability to descend into into 
the properties of complex objects to test assertions.

* Low overhead for writing tests; it should be easy to add tests with 
minimal overhead per testcase (I believe this implies multiple tests per 

* No weird codepaths (aka the "no eval" requirement); The harness should 
run code in a context that it would typically be run by end users 
(except when explicitly testing eval), and should avoid requiring 
contexts with unexpected rules for e.g. variable scope.

* No browser specific codepaths

* Support for asynchronous tests; many DOM methods are asynchronous and 
therefore a reliable way of testing asynchronous behaviour e.g. that a 
given event is fired, is needed. This also implies that it must be 
possible to time out tests it they fail to produce a result.

* Avoid unnecessary complexity: The test harness should be as simple as 
possible whilst meeting the other requirements. In particular it should 
itself rely on as few DOM features as possible (but could assume a 
mostly-functional ECMAScript engine).

* Ability to run single test multiple times in a single run?: this is 
needed if we want tests to be compiled in typical modern ECMAScript 
engines, which can cause behaviour changes.

* Metadata? There are possibilities here e.g. marking a test as 
"expected to fail if other test x also fails"

Received on Tuesday, 20 April 2010 15:48:58 UTC