Re: Make SCXML IRP tests more CI-able

I have converted the rest of the tests in Stefan's original email to use 
conf:delay.  I may have screwed something  up so check carefully the 
first time  you run them.

- Jim
On 6/28/2014 5:05 PM, Stefan Radomski wrote:
> Works great! For ECMAScript the following substitutions in 
> confECMA.xsl will reduce all delay in test175 to 1/100th:
>
> <!-- delayexpr takes the value of the specified variable -->
> <xsl:template match="//@conf:delayFromVar">
> <xsl:attribute name="delayexpr">(Var<xsl:value-of select="." 
> />.slice(0, - 1)) * 100 + 'ms'</xsl:attribute>
> </xsl:template>
>
> <!-- computes a delayexpr based on the value passed in.  this lets 
> platforms determine how long to delay timeout
> events which cause the test to fail.  The default value provided here 
> is pretty long -->
> <xsl:template match="//@conf:delay">
> <xsl:attribute name="delayexpr">'<xsl:value-of select=". * 
> 100"/>ms'</xsl:attribute>
> </xsl:template>
>
>
> On Jun 28, 2014, at 18:43, Jim Barnett <1jhbarnett@gmail.com 
> <mailto:1jhbarnett@gmail.com>> wrote:
>
>> I have added conf:delay that takes a relative value.  The default 
>> implementation in confECMA just appends  's' to it (so 
>> conf:delay=".5" turns into delayexpr="'.5s'".  I have converted 
>> test175 to use this element.
>>
>> However, I haven't been able to test the results because the pyscxml 
>> console (which I normally use for testing) is offline.  So I have 
>> just checked in these changes until I can be sure that I haven't 
>> screwed things up.  Once it's clear that conf:delay is  working ok, 
>> I'll convert the other tests.
>>
>> test175 is an odd test case, because it tests that delayexpr can be 
>> taken from a variable, so there is still a 1s delay hard-coded into 
>> the test.  There should be more of a speed-up in other tests (or I 
>> can reduce the hardcoded delay to .5s and then set conf:delay=".25".
>>
>> Let me know what you think,
>>
>> - Jim
>>
>> On 6/28/2014 11:38 AM, Gavin Kistner wrote:
>>> Whoops, sent that too fast. I was going to add:
>>>
>>> However, I’d prefer to remove this hack and have the tests generated 
>>> with reasonable values. I support Stefan’s suggestion of having the 
>>> value of the delay be relative, e.g. conf:delay=“1” vs conf:delay=“2”.
>>>
>>> On Jun 28, 2014, at 9:35 AM, Gavin Kistner <phrogz@me.com 
>>> <mailto:phrogz@me.com>> wrote:
>>>
>>>> FWIW, my “solution” is that my test runner waits until there are no 
>>>> more events to process peeks at the next delayed event off the 
>>>> queue, and manually advances the time for the state machine by the 
>>>> necessary amount so that it will be dequeued. My insanely slow, 
>>>> unoptimized runtime is currently processing the auto tests in about 
>>>> 2 seconds.
>>>>
>>>> On Jun 28, 2014, at 9:09 AM, Stefan Radomski 
>>>> <radomski@tk.informatik.tu-darmstadt.de 
>>>> <mailto:radomski@tk.informatik.tu-darmstadt.de>> wrote:
>>>>
>>>>> Can’t you keep the relative delay values and introduce a factor by 
>>>>> which implementations can multiply these? This way, the performant 
>>>>> implementations can set the factor small and others keep it at 1 
>>>>> or even bigger.
>>>>>
>>>>> If it’s too much of a hassle, just keep it as it is and I will 
>>>>> live with a 1 minute delay every time the tests run.
>>>>>   Stefan
>>>>>
>>>>> On Jun 28, 2014, at 16:10, Jim Barnett <jim.barnett@genesys.com 
>>>>> <mailto:jim.barnett@genesys.com>> wrote:
>>>>>
>>>>>> One problem that occurs to me is that we may want different 
>>>>>> timeout values for different tests (some tests involve <invoke>, 
>>>>>> which is going to be slower than other operations).  So setting a 
>>>>>> single value forconf:delaymay not work well.  We could just say 
>>>>>> that platforms can adjust the timeout values to suit their 
>>>>>> conditions, but that requires a lot of editing for each 
>>>>>> platform.  Would it be acceptable for each platform to 
>>>>>> setconf:delayto the largest timeout value it would ever need?
>>>>>> -Jim
>>>>>> *From:*Jim Barnett [mailto:1jhbarnett@gmail.com]
>>>>>> *Sent:*Saturday, June 28, 2014 8:35 AM
>>>>>> *To:*www-voice@w3.org <mailto:www-voice@w3.org>
>>>>>> *Subject:*Re: Make SCXML IRP tests more CI-able
>>>>>> I wanted to make sure that the timeout didn't fire too soon since 
>>>>>> that would cause the test to fail when it should succeed. Maybe I 
>>>>>> should addconf:timeOutand let each platform set the value as it 
>>>>>> sees fit.
>>>>>>
>>>>>> - Jim
>>>>>> On 6/28/2014 8:09 AM, Stefan Radomski wrote:
>>>>>>
>>>>>>     Hey there,
>>>>>>     we’d like to run the non-manual tests as part of continuous
>>>>>>     integration. As such, we’d be happy if we could reduce their
>>>>>>     runtime somewhat. Worst offenders are:
>>>>>>     ecma/test175.scxml     =   3.07 sec
>>>>>>     ecma/test185.scxml     =   2.07 sec
>>>>>>     ecma/test186.scxml     =   2.07 sec
>>>>>>     ecma/test187.scxml     =  10.07 sec
>>>>>>     ecma/test207.scxml     =   3.10 sec
>>>>>>     ecma/test208.scxml     =   5.07 sec
>>>>>>     ecma/test210.scxml     =   5.07 sec
>>>>>>     ecma/test236.scxml     =   2.07 sec
>>>>>>     ecma/test237.scxml     =   3.07 sec
>>>>>>     ecma/test252.scxml     =   2.07 sec
>>>>>>     ecma/test409.scxml     =   1.07 sec
>>>>>>     ecma/test422.scxml     =   5.09 sec
>>>>>>     ecma/test423.scxml     =   1.07 sec
>>>>>>     ecma/test553.scxml     =   3.07 sec
>>>>>>     ecma/test554.scxml     =   2.08 sec
>>>>>>     ecma/test579.scxml     =   2.07 sec
>>>>>>     Total Test time (real) =  65.79 sec
>>>>>>     Can we reduce the various delay expressions in these tests?
>>>>>>     I’d vote to cut them all to one tenth of their current
>>>>>>     values. That is 1s becomes 100ms - I can’t imagine that there
>>>>>>     is an implementation out there that takes more than a
>>>>>>     millisecond to process a benign macrostep and it cuts down
>>>>>>     total time for all automated tests to appr. 1/4th.
>>>>>>     Total Test time (real) =  18.83 sec
>>>>>>     It’s not exactly critical to do, but considering how often I
>>>>>>     run the ECMA tests it more than justified this mail.
>>>>>>     Regards
>>>>>>     Stefan
>>>>>>
>>>>>> --
>>>>>> Jim Barnett
>>>>>> Genesys
>>>>>
>>>>
>>>
>>
>> -- 
>> Jim Barnett
>> Genesys
>

-- 
Jim Barnett
Genesys