Re: Make SCXML IRP tests more CI-able

I have added conf:delay that takes a relative value.  The default 
implementation in confECMA just appends  's' to it (so conf:delay=".5" 
turns into delayexpr="'.5s'".  I have converted test175 to use this 
element.

However, I haven't been able to test the results because the pyscxml 
console (which I normally use for testing) is offline.  So I have just 
checked in these changes until I can be sure that I haven't screwed 
things up.  Once it's clear that conf:delay is  working ok, I'll convert 
the other tests.

test175 is an odd test case, because it tests that delayexpr can be 
taken from a variable, so there is still a 1s delay hard-coded into the 
test.  There should be more of a speed-up in other tests (or I can 
reduce the hardcoded delay to .5s and then set conf:delay=".25".

Let me know what you think,

- Jim

On 6/28/2014 11:38 AM, Gavin Kistner wrote:
> Whoops, sent that too fast. I was going to add:
>
> However, I’d prefer to remove this hack and have the tests generated 
> with reasonable values. I support Stefan’s suggestion of having the 
> value of the delay be relative, e.g. conf:delay=“1” vs conf:delay=“2”.
>
> On Jun 28, 2014, at 9:35 AM, Gavin Kistner <phrogz@me.com 
> <mailto:phrogz@me.com>> wrote:
>
>> FWIW, my “solution” is that my test runner waits until there are no 
>> more events to process peeks at the next delayed event off the queue, 
>> and manually advances the time for the state machine by the necessary 
>> amount so that it will be dequeued. My insanely slow, unoptimized 
>> runtime is currently processing the auto tests in about 2 seconds.
>>
>> On Jun 28, 2014, at 9:09 AM, Stefan Radomski 
>> <radomski@tk.informatik.tu-darmstadt.de 
>> <mailto:radomski@tk.informatik.tu-darmstadt.de>> wrote:
>>
>>> Can’t you keep the relative delay values and introduce a factor by 
>>> which implementations can multiply these? This way, the performant 
>>> implementations can set the factor small and others keep it at 1 or 
>>> even bigger.
>>>
>>> If it’s too much of a hassle, just keep it as it is and I will live 
>>> with a 1 minute delay every time the tests run.
>>>   Stefan
>>>
>>> On Jun 28, 2014, at 16:10, Jim Barnett <jim.barnett@genesys.com 
>>> <mailto:jim.barnett@genesys.com>> wrote:
>>>
>>>> One problem that occurs to me is that we may want different timeout 
>>>> values for different tests (some tests involve <invoke>, which is 
>>>> going to be slower than other operations).  So setting a single 
>>>> value forconf:delaymay not work well.  We could just say that 
>>>> platforms can adjust the timeout values to suit their conditions, 
>>>> but that requires a lot of  editing for each platform.  Would it be 
>>>> acceptable for each platform to setconf:delayto the largest timeout 
>>>> value it would ever need?
>>>> -Jim
>>>> *From:*Jim Barnett [mailto:1jhbarnett@gmail.com]
>>>> *Sent:*Saturday, June 28, 2014 8:35 AM
>>>> *To:*www-voice@w3.org <mailto:www-voice@w3.org>
>>>> *Subject:*Re: Make SCXML IRP tests more CI-able
>>>> I wanted to make sure that the timeout didn't fire too soon since 
>>>> that would cause the test to fail when it should succeed.  Maybe I 
>>>> should addconf:timeOutand let each platform set the value as it 
>>>> sees fit.
>>>>
>>>> - Jim
>>>> On 6/28/2014 8:09 AM, Stefan Radomski wrote:
>>>>
>>>>     Hey there,
>>>>     we’d like to run the non-manual tests as part of continuous
>>>>     integration. As such, we’d be happy if we could reduce their
>>>>     runtime somewhat. Worst offenders are:
>>>>     ecma/test175.scxml     =   3.07 sec
>>>>     ecma/test185.scxml     =   2.07 sec
>>>>     ecma/test186.scxml     =   2.07 sec
>>>>     ecma/test187.scxml     =  10.07 sec
>>>>     ecma/test207.scxml     =   3.10 sec
>>>>     ecma/test208.scxml     =   5.07 sec
>>>>     ecma/test210.scxml     =   5.07 sec
>>>>     ecma/test236.scxml     =   2.07 sec
>>>>     ecma/test237.scxml     =   3.07 sec
>>>>     ecma/test252.scxml     =   2.07 sec
>>>>     ecma/test409.scxml     =   1.07 sec
>>>>     ecma/test422.scxml     =   5.09 sec
>>>>     ecma/test423.scxml     =   1.07 sec
>>>>     ecma/test553.scxml     =   3.07 sec
>>>>     ecma/test554.scxml     =   2.08 sec
>>>>     ecma/test579.scxml     =   2.07 sec
>>>>     Total Test time (real) =  65.79 sec
>>>>     Can we reduce the various delay expressions in these tests? I’d
>>>>     vote to cut them all to one tenth of their current values. That
>>>>     is 1s becomes 100ms - I can’t imagine that there is an
>>>>     implementation out there that takes more than a millisecond to
>>>>     process a benign macrostep and it cuts down total time for all
>>>>     automated tests to appr. 1/4th.
>>>>     Total Test time (real) =  18.83 sec
>>>>     It’s not exactly critical to do, but considering how often I
>>>>     run the ECMA tests it more than justified this mail.
>>>>     Regards
>>>>     Stefan
>>>>
>>>> --
>>>> Jim Barnett
>>>> Genesys
>>>
>>
>

-- 
Jim Barnett
Genesys