Re: Test timeouts

On 17/09/13 15:20, Bjoern Hoehrmann wrote:
> * James Graham wrote:
>> In previous discussions there has been some contention about how to set
>> up the timeout in the first place, and how much author control there
>> should be. At the moment there is full author control — i.e. a precise
>> timeout in ms — via the setup() function in testharness.js. This
>> approach has one critical disadvantage; it is hard for external tools to
>> read the timeout. Since you might hang the whole browser it certainly
>> needs to be possible for the test runner to kill the whole process after
>> a suitable delay that must be at least as long as the test timeout.
>
> It seems strange to bother authors with this. I would expect the system
> to use data from past runs to detect and deal with abnormalities. In a
> system that also tests for performance issues you would do that anyway.
>

Typically regression harnesses don't also deal with performance issues.

I suppose for automation it would be possible to gather expected 
runtimes during the test import phase, at the cost of potentially making 
this rather slow, and then use that to set the expectations for 
subsequent runs. I will have to think more about this. But we still need 
something for the case where a user just loads the test in their browser.

Received on Tuesday, 17 September 2013 14:33:57 UTC