RE: Test harness requirements

Thanks for submitting, in general this seems to be a good direction and will enable the group to create asynchronous api tests.
As such I have incorporated this into the existing domtestcase.js library file for general usage by the group.

Though I did not move it to the approved folder since this should only be done with consensus from the group.

Additionally I have made a few changes to the library
   * Add a few more 'tests' - set a custom timeout for example
   * Removed the assumption that object.name exists (not standard)
   * Output to a table rather than a single div
   * Un-rolled a lot of the code so that it is a lot more readable
   * Removed the non-standard object.name and set the name in the expose calls 

Other changes that could be done
  * Add some indication that the test(s) are running - Loading...
  * Add a way to mark a test Not Implemented rather than Pass or Fail

Folks can load it up and diff the changes from james post , or run it from the following location from the w3c

http://dvcs.w3.org/hg/html/raw-file/tip/tests/submission/Microsoft/interface/apisample.htm


If you have feeback please respond or we can discuss at next weeks IRC/Conf call.

-Kris

-----Original Message-----
From: James Graham [mailto:jgraham@opera.com] 
Sent: Tuesday, May 18, 2010 2:27 AM
To: Geoffrey Sneddon
Cc: Kris Krueger; Anne van Kesteren; public-html-testsuite@w3.org
Subject: Re: Test harness requirements

On 05/04/2010 04:09 PM, Geoffrey Sneddon wrote:

> As was said on the previous telecon, neither myself nor James have any 
> objection to approving the test as is; however, I still think we do 
> need to design the harness to deal with everything it needs to do 
> initially, as a number of test harness really suffer from having 
> things like support for async tests hacked on to them at a later date. 
> This will mean moving tests already approved over to a new harness at 
> a later date, and that we probably don't want to approve large test 
> suites until we know what the API will be.


So on the subject of test harnesses, I sketched out an idea-as code [1]. 
It is not supposed to be complete; in particular it should have more
assert_* functions and better output (including the source of the test steps, for example). There is also an example test document showing roughly how the harness works. Hopefully that is self-explanatory but, in case it is not:

The harness exposes a small number of global functions, the most important being "test" and "test_async". The test function creates and runs a test synchronously. It takes a function that contains the test code, and a name for the test. test_async allows for asynchronous tests, it takes the name of a test and returns a test object. The "step" method on that test object can be called at any later time taking a function that runs further assertions for this test. Once the test is complete (no more steps will be run), the "done" method on the test object must be called. If the done method is not called a certain time after the test starts then it times out. All calls to test and test_async must occur before the load event fires.

Assertions are provided by assert_* functions. At the moment assert_true, assert_equals, assert_object_equals and assert_exists. More are needed to cover common things like checking that a function throws, and checking various IDL properties (e.g. assert_readonly).

To enable integration with browser regression test systems, there are a couple of functions for registering results handlers; add_results_callback and add_completion_callback. One is called once per result, the other is called when all results are available.

Hopefully that is enough information to start some discussion about strengths and weaknesses so we can iterate in the direction of a good design.

[1] http://hg.hoppipolla.co.uk/hgwebdir.cgi/domharness/

Received on Thursday, 10 June 2010 04:50:35 UTC