Re: Make SCXML IRP tests more CI-able

Whoops, sent that too fast. I was going to add:

However, I’d prefer to remove this hack and have the tests generated with reasonable values. I support Stefan’s suggestion of having the value of the delay be relative, e.g. conf:delay=“1” vs conf:delay=“2”.

On Jun 28, 2014, at 9:35 AM, Gavin Kistner <phrogz@me.com> wrote:

> FWIW, my “solution” is that my test runner waits until there are no more events to process peeks at the next delayed event off the queue, and manually advances the time for the state machine by the necessary amount so that it will be dequeued. My insanely slow, unoptimized runtime is currently processing the auto tests in about 2 seconds.
>  
> On Jun 28, 2014, at 9:09 AM, Stefan Radomski <radomski@tk.informatik.tu-darmstadt.de> wrote:
> 
>> Can’t you keep the relative delay values and introduce a factor by which implementations can multiply these? This way, the performant implementations can set the factor small and others keep it at 1 or even bigger.
>> 
>> If it’s too much of a hassle, just keep it as it is and I will live with a 1 minute delay every time the tests run.
>>   Stefan
>> 
>> On Jun 28, 2014, at 16:10, Jim Barnett <jim.barnett@genesys.com> wrote:
>> 
>>> One problem that occurs to me is that we may want different timeout values for different tests (some tests involve <invoke>, which is going to be slower than other operations).  So setting a single value for conf:delay may not work well.  We could just say that platforms can adjust the timeout values to suit their conditions, but that requires a lot of  editing for each platform.  Would it be acceptable for each platform to set conf:delay to the largest timeout value it would ever need?
>>>  
>>> -          Jim
>>>  
>>> From: Jim Barnett [mailto:1jhbarnett@gmail.com] 
>>> Sent: Saturday, June 28, 2014 8:35 AM
>>> To: www-voice@w3.org
>>> Subject: Re: Make SCXML IRP tests more CI-able
>>>  
>>> I wanted to make sure that the timeout didn't fire too soon since that would cause the test to fail when it should succeed.  Maybe I should add conf:timeOut and let each platform set the value as it sees fit.  
>>> 
>>> - Jim
>>> On 6/28/2014 8:09 AM, Stefan Radomski wrote:
>>> Hey there,
>>>  
>>> we’d like to run the non-manual tests as part of continuous integration. As such, we’d be happy if we could reduce their runtime somewhat. Worst offenders are:
>>>  
>>> ecma/test175.scxml     =   3.07 sec
>>> ecma/test185.scxml     =   2.07 sec
>>> ecma/test186.scxml     =   2.07 sec
>>> ecma/test187.scxml     =  10.07 sec
>>> ecma/test207.scxml     =   3.10 sec
>>> ecma/test208.scxml     =   5.07 sec
>>> ecma/test210.scxml     =   5.07 sec
>>> ecma/test236.scxml     =   2.07 sec
>>> ecma/test237.scxml     =   3.07 sec
>>> ecma/test252.scxml     =   2.07 sec
>>> ecma/test409.scxml     =   1.07 sec
>>> ecma/test422.scxml     =   5.09 sec
>>> ecma/test423.scxml     =   1.07 sec
>>> ecma/test553.scxml     =   3.07 sec
>>> ecma/test554.scxml     =   2.08 sec
>>> ecma/test579.scxml     =   2.07 sec
>>>  
>>> Total Test time (real) =  65.79 sec
>>>  
>>> Can we reduce the various delay expressions in these tests? I’d vote to cut them all to one tenth of their current values. That is 1s becomes 100ms - I can’t imagine that there is an implementation out there that takes more than a millisecond to process a benign macrostep and it cuts down total time for all automated tests to appr. 1/4th.
>>>  
>>> Total Test time (real) =  18.83 sec
>>>  
>>> It’s not exactly critical to do, but considering how often I run the ECMA tests it more than justified this mail.
>>>  
>>> Regards
>>> Stefan
>>>  
>>> -- 
>>> Jim Barnett
>>> Genesys
>> 
>