Re: Make SCXML IRP tests more CI-able

Can’t you keep the relative delay values and introduce a factor by which implementations can multiply these? This way, the performant implementations can set the factor small and others keep it at 1 or even bigger.

If it’s too much of a hassle, just keep it as it is and I will live with a 1 minute delay every time the tests run.
  Stefan

On Jun 28, 2014, at 16:10, Jim Barnett <jim.barnett@genesys.com<mailto:jim.barnett@genesys.com>> wrote:

One problem that occurs to me is that we may want different timeout values for different tests (some tests involve <invoke>, which is going to be slower than other operations).  So setting a single value for conf:delay may not work well.  We could just say that platforms can adjust the timeout values to suit their conditions, but that requires a lot of  editing for each platform.  Would it be acceptable for each platform to set conf:delay to the largest timeout value it would ever need?

-          Jim

From: Jim Barnett [mailto:1jhbarnett@gmail.com]
Sent: Saturday, June 28, 2014 8:35 AM
To: www-voice@w3.org<mailto:www-voice@w3.org>
Subject: Re: Make SCXML IRP tests more CI-able

I wanted to make sure that the timeout didn't fire too soon since that would cause the test to fail when it should succeed.  Maybe I should add conf:timeOut and let each platform set the value as it sees fit.

- Jim
On 6/28/2014 8:09 AM, Stefan Radomski wrote:
Hey there,

we’d like to run the non-manual tests as part of continuous integration. As such, we’d be happy if we could reduce their runtime somewhat. Worst offenders are:

ecma/test175.scxml     =   3.07 sec
ecma/test185.scxml     =   2.07 sec
ecma/test186.scxml     =   2.07 sec
ecma/test187.scxml     =  10.07 sec
ecma/test207.scxml     =   3.10 sec
ecma/test208.scxml     =   5.07 sec
ecma/test210.scxml     =   5.07 sec
ecma/test236.scxml     =   2.07 sec
ecma/test237.scxml     =   3.07 sec
ecma/test252.scxml     =   2.07 sec
ecma/test409.scxml     =   1.07 sec
ecma/test422.scxml     =   5.09 sec
ecma/test423.scxml     =   1.07 sec
ecma/test553.scxml     =   3.07 sec
ecma/test554.scxml     =   2.08 sec
ecma/test579.scxml     =   2.07 sec

Total Test time (real) =  65.79 sec

Can we reduce the various delay expressions in these tests? I’d vote to cut them all to one tenth of their current values. That is 1s becomes 100ms - I can’t imagine that there is an implementation out there that takes more than a millisecond to process a benign macrostep and it cuts down total time for all automated tests to appr. 1/4th.

Total Test time (real) =  18.83 sec

It’s not exactly critical to do, but considering how often I run the ECMA tests it more than justified this mail.

Regards
Stefan

--
Jim Barnett
Genesys