Re: Recent testharness.js updates causing issues with webperf tests

>> It appears that a recent update to testharness.js (and thus published
>> on test.w3.org) may be causing some issues with some of our tests.  It
>> looks like asserts() that happen after the load handler for the page
>> are not getting registered in the summary of Pass/Fail results.
>>
>>
>> For example, test.w3.org with the new changes shows "Found 0 tests" in
>> IE and Chrome while the older version on w3c-test.org works properly
>> and shows all of the Pass/Fail results:
>>
>> http://test.w3.org/webperf/tests/approved/test_navigate_within_document.htm
>>
>> http://w3c-test.org/webperf/tests/approved/test_navigate_within_document.htm

I'm surprised that ever worked; it wasn't supposed to :) I suspect that 
what is happening now is that the harness is being stricter about not 
responding to stuff that happens after it first thinks that it's complete.

In general the test harness has no way of knowing whether more tests 
will be defined at some future time or not. If there are no pending 
tests when load fires it will assume that there won't be any added in 
the future.

There are approaches to overcome this; the most appropriate approach, or 
combination of approaches, will depend on your particular situation:

1) Define all your tests upfront (before load) and later call .step on 
the predefined test object. This has the nice property that the number 
of tests should be consistent between runs even if browsers have some 
bug that prevents them from running all tests.

2) Set the test harness to wait until you explicitly indicate that all 
the tests are done. To do this use the setup() function sometime before 
the load event as so:
setup({explicit_done:true})
then later, once there are no more tests to define, call
done()

Received on Thursday, 17 March 2011 16:05:59 UTC