Re: Make SCXML IRP tests more CI-able

Hey there,

the tests work fine for confECMA.xsl. By substituting the respective lines via sed:

sed -ie "s/<xsl:attribute name=\"delayexpr\">Var<xsl:value-of select=\".\" \/><\/xsl:attribute>/<xsl:attribute name=\"delayexpr\">(Var<xsl:value-of select=\".\" \/>.slice(0, - 1)) * 100 + 'ms'<\/xsl:attribute>/" confEcma.xsl
sed -ie "s/<xsl:attribute name=\"delayexpr\">'<xsl:value-of select=\".\"\/>s'<\/xsl:attribute>/<xsl:attribute name=\"delayexpr\">'<xsl:value-of select=\". * 100\"\/>ms'<\/xsl:attribute>/" confEcma.xsl

I got a very nice speed-up

Total Test time (real) =  16.08 sec

But the confXPath.xsl transformations are broken now. For every transformation I get:

Warning: on line 250 of confXPath.xsl:
  The attribute axis starting at an attribute node will never select anything

Respective line is:

<xsl:attribute name="delayexpr">'<xsl:value-of select="@id"/>s'</xsl:attribute>

I got rid of the errors by substituting ‘@id' with ‘.’ as in the confEcma.xsl, but I cannot tell whether this is correct as we do not run the xpath tests.

Regards
Stefan

On Jun 29, 2014, at 4:00, Jim Barnett <1jhbarnett@gmail.com<mailto:1jhbarnett@gmail.com>> wrote:

I have converted the rest of the tests in Stefan's original email to use conf:delay.  I may have screwed something  up so check carefully the first time  you run them.

- Jim
On 6/28/2014 5:05 PM, Stefan Radomski wrote:
Works great! For ECMAScript the following substitutions in confECMA.xsl will reduce all delay in test175 to 1/100th:

<!-- delayexpr takes the value of the specified variable -->
<xsl:template match="//@conf:delayFromVar">
<xsl:attribute name="delayexpr">(Var<xsl:value-of select="." />.slice(0, - 1)) * 100 + 'ms'</xsl:attribute>
</xsl:template>

<!-- computes a delayexpr based on the value passed in.  this lets platforms determine how long to delay timeout
events which cause the test to fail.  The default value provided here is pretty long -->
<xsl:template match="//@conf:delay">
<xsl:attribute name="delayexpr">'<xsl:value-of select=". * 100"/>ms'</xsl:attribute>
</xsl:template>


On Jun 28, 2014, at 18:43, Jim Barnett <1jhbarnett@gmail.com<mailto:1jhbarnett@gmail.com>> wrote:

I have added conf:delay that takes a relative value.  The default implementation in confECMA just appends  's' to it (so conf:delay=".5" turns into delayexpr="'.5s'".  I have converted test175 to use this element.

However, I haven't been able to test the results because the pyscxml console (which I normally use for testing) is offline.  So I have just checked in these changes until I can be sure that I haven't screwed things up.  Once it's clear that conf:delay is  working ok, I'll convert the other tests.

test175 is an odd test case, because it tests that delayexpr can be taken from a variable, so there is still a 1s delay hard-coded into the test.  There should be more of a speed-up in other tests (or I can reduce the hardcoded delay to .5s and then set conf:delay=".25".

Let me know what you think,

- Jim

On 6/28/2014 11:38 AM, Gavin Kistner wrote:
Whoops, sent that too fast. I was going to add:

However, I’d prefer to remove this hack and have the tests generated with reasonable values. I support Stefan’s suggestion of having the value of the delay be relative, e.g. conf:delay=“1”<conf:delay=%E2%80%9C1%E2%80%9D> vs conf:delay=“2”<conf:delay=%E2%80%9C2%E2%80%9D>.

On Jun 28, 2014, at 9:35 AM, Gavin Kistner <phrogz@me.com<mailto:phrogz@me.com>> wrote:

FWIW, my “solution” is that my test runner waits until there are no more events to process peeks at the next delayed event off the queue, and manually advances the time for the state machine by the necessary amount so that it will be dequeued. My insanely slow, unoptimized runtime is currently processing the auto tests in about 2 seconds.

On Jun 28, 2014, at 9:09 AM, Stefan Radomski <radomski@tk.informatik.tu-darmstadt.de<mailto:radomski@tk.informatik.tu-darmstadt.de>> wrote:

Can’t you keep the relative delay values and introduce a factor by which implementations can multiply these? This way, the performant implementations can set the factor small and others keep it at 1 or even bigger.

If it’s too much of a hassle, just keep it as it is and I will live with a 1 minute delay every time the tests run.
  Stefan

On Jun 28, 2014, at 16:10, Jim Barnett <jim.barnett@genesys.com<mailto:jim.barnett@genesys.com>> wrote:

One problem that occurs to me is that we may want different timeout values for different tests (some tests involve <invoke>, which is going to be slower than other operations).  So setting a single value for conf:delay may not work well.  We could just say that platforms can adjust the timeout values to suit their conditions, but that requires a lot of  editing for each platform.  Would it be acceptable for each platform to set conf:delay to the largest timeout value it would ever need?

-          Jim

From: Jim Barnett [mailto:1jhbarnett@gmail.com]
Sent: Saturday, June 28, 2014 8:35 AM
To: www-voice@w3.org<mailto:www-voice@w3.org>
Subject: Re: Make SCXML IRP tests more CI-able

I wanted to make sure that the timeout didn't fire too soon since that would cause the test to fail when it should succeed.  Maybe I should add conf:timeOut and let each platform set the value as it sees fit.

- Jim
On 6/28/2014 8:09 AM, Stefan Radomski wrote:
Hey there,

we’d like to run the non-manual tests as part of continuous integration. As such, we’d be happy if we could reduce their runtime somewhat. Worst offenders are:

ecma/test175.scxml     =   3.07 sec
ecma/test185.scxml     =   2.07 sec
ecma/test186.scxml     =   2.07 sec
ecma/test187.scxml     =  10.07 sec
ecma/test207.scxml     =   3.10 sec
ecma/test208.scxml     =   5.07 sec
ecma/test210.scxml     =   5.07 sec
ecma/test236.scxml     =   2.07 sec
ecma/test237.scxml     =   3.07 sec
ecma/test252.scxml     =   2.07 sec
ecma/test409.scxml     =   1.07 sec
ecma/test422.scxml     =   5.09 sec
ecma/test423.scxml     =   1.07 sec
ecma/test553.scxml     =   3.07 sec
ecma/test554.scxml     =   2.08 sec
ecma/test579.scxml     =   2.07 sec

Total Test time (real) =  65.79 sec

Can we reduce the various delay expressions in these tests? I’d vote to cut them all to one tenth of their current values. That is 1s becomes 100ms - I can’t imagine that there is an implementation out there that takes more than a millisecond to process a benign macrostep and it cuts down total time for all automated tests to appr. 1/4th.

Total Test time (real) =  18.83 sec

It’s not exactly critical to do, but considering how often I run the ECMA tests it more than justified this mail.

Regards
Stefan

--
Jim Barnett
Genesys




--
Jim Barnett
Genesys


--
Jim Barnett
Genesys