Re: April CCXML: test case conflicts with ECMA rules - ISSUE-677

Petr,

Sure, let's talk CPU metrics of multiple small scripts doing the same
work as a single big script.

The metrics are the output of the "time" system call
on Linux, which reports the amount of elapsed time, user
CPU time, and system CPU time against a simple program
using a javascript engine.

Each test "runs the script" 10,000 times.

test 1:
=====
the script is:
var x = new Object();
var y = 7;
var z = 3;

30,000 small scripts (line by line model)
0.720u 0.012s 0:00.74

10,000 big scripts (all at once model)
0.364u 0.008s 0:00.38

test 2:
======
the script is:
var x = new Object();
var y = 7;
var z = 3;
var a = new Object();
var b = 7;
var c = y;


60,000 small scripts (line by line model)
1.416u 0.004s 0:01.45

10,000 big scripts (all at once model)
0.524u 0.004s 0:00.53

Conclusions:
============
The first test shows the same amount of work
taking twice as long using small scripts vs  big scripts.

The second test doubles the amount of ECMA instructions
when compared to test 1,
but shows that small scripts take about 3 times as long
as big scripts. Note that as more ECMA work is added, the
savings of big vs. small increase.

We see these results duplicated in our CCXML browser, and can run 2x,3x, 
or more - channels of big script CCXML vs
little script CCXML.

Regards,
Chris

Petr Kuba wrote:
> Chris,
>
> When you mention performance issues do you have some numbers showing 
> the difference? E.g. how many concurrent CCXML sessions with large 
> ECMA scripts are you able to run compared to CCXML sessions with small 
> ECMA scripts containing the same code?
>
> Thanks,
> Petr
>
>
> On 26.4.2010 20:28, Chris Davis wrote:
>> RJ,
>>
>> Is it stated somewhere in CCXML that these elements should be
>> "executed and fed...one at a time" to ECMA?
>>
>> Here are some comments on behavior and performance, then some 
>> suggestions:
>>
>> BEHAVIOR
>> ^^^^^^^^
>> Consider the following C program fragment and note the comments:
>> --------------------- example 1 ----------
>> int id = 1;
>>
>> void foo()
>> {
>>      // here we see the global id = 1
>>
>>      if (id==1) // this will pass
>>      {
>>          int id = 2; // id=2 only true until the }
>>      }
>>
>>      // id still = 1 here
>> }
>> ---------------------------------------------
>> Now consider a similar fragment in ECMA:
>> ------------------------ example 2 ---------------------
>> var id = 1;
>> function foo()
>> {
>>      // id = undefined here due to ECMA scope rules
>>      if( id == 1) // never passes
>>      {
>>          var id = 2; // causes id to be set to undefined in current
>> scope before execution
>>      }
>>
>>      // id is undefined here
>> }
>> ---------------------------------------------
>> Now consider the effects of line-by-line processing
>> in CCXML:
>> ------------------------ example 3 ---------------------
>> <var name="id" expr = "1"/>
>> <transition>
>> <if cond="id == 1"> <!-- passes because ECMA not exposed yet to the var
>> declaration -->
>> <var name="id" expr="2"/> <!-- causes id to be set to 2 in current
>> scope  -->
>> </if>
>>
>> <!-- id is 2 here because we now have set id in the current scope, 
>> but is
>>           only "here" if above if check ran -->
>>
>> </transition>
>>
>> ---------------------------------------------
>> Note that example 3 has different behavior than both C (example 1) and
>> ECMA(example 2).
>>
>> This was the example similar to test case 798. Consider a more hideous
>> side effect
>> of "line by line" processing:
>>
>> --------------- example 4 ----------------------
>> <transition>
>> <if cond="something == somethingelse">
>> <var name="x" expr="1"/>
>> <!-- x now "lives" here and till end of transition -->
>> </if>
>>
>> <if cond="1==x"> <!-- will throw ECMA error if something !=
>> somethingelse -->
>> </if>
>> </transition>
>> -----------------------------------------------
>> If example 4 is instead treated using ECMA rules (example 2), x will be
>> at the most
>> undefined and at least *not* throw an ECMA error.
>>
>> We look at CCXML's stated use of EMCA for scope rules, variable 
>> resolution,
>> etc. and see that CCXML is built on ECMA. It is surprising then that the
>> CCXML tags
>> "var", "if" etc. all are supposed
>> to behave differently(example 3 above) from an ECMA script
>> itself(example 2 above). These behaviors
>> are different despite operating in an ECMA scope and having the same
>> names(<if>, <var>) as javascript syntax itself.
>>
>> Was it really the intention of CCXML to create this unique behavior?
>>
>> PERFORMANCE
>> ^^^^^^^^^^^
>> Any requirement to run many small
>> scripts for line-by-line processing as opposed to a small number of
>> large scripts
>> would have a negative impact on performance, and in multiple places 
>> CCXML
>> seems designed for performance:
>>
>>      "Special attention has been paid to constraining
>>      ECMAScript features that require proportionately
>>      large amounts of system memory, and continuous or
>>      proportionately large amounts of processing power."
>>
>>      "Because CCXML and ECMAScript applications can be
>>      CPU intensive to compile we define <script>'s src
>>      attribute (defining the URI of the document to load)
>>      to be a static string instead of a dynamically
>>      valued ECMAScript result"
>>
>> If performance is a concern then it is surprising that the
>> Recommendation forces
>> the most CPU intensive of all operations: execution of many tiny scripts
>> instead
>> of a few large ones (the line by line model).
>>
>> In 2003 we traded emails with Brendan Eich - our complaint to him was
>> "javascript is slow and CPU intensive". He claimed it was not, and
>> pointed out that our excessive use of
>> large numbers of tiny scripts was not as efficient as
>> a small number of large scripts (same amount of work but called
>> differently).
>>
>> We ran some tests and confirmed what he said.
>> As a result we dumped the open source "OpenVXI" for our VoiceXML
>> browser and fashioned a different way to integrate ECMA - something we
>> carry over into our CCXML browser.
>>
>> Finally, we fear that mandating inefficient ECMA processing in the
>> Recommendation will leave
>> CCXML browsers out in the cold. Recent improvements in ECMA engines such
>> as TraceMonkey
>> and Google V8 excel at processing large scripts but users of tiny
>> scripts(like CCXML browsers) will see
>> less improvement.
>>
>> SUGGESTIONS
>> ^^^^^^^^^^^
>> We'd like to see the test case changed or tossed to not require "type 3
>> behavior".
>>
>> If "type 3 behavior" is required we'd suggest a section in
>> the Recommendation explaining the affliction, preferably with the
>> dangers/side effects(example 4).
>>
>> Regards,
>> Chris
>>
>> RJ Auburn wrote:
>>> Chris and Petr:
>>>
>>> This is tracked as ISSUE-677.
>>>
>>> It is the intent that each of the executable content items (var, 
>>> assign, script and so on) should be executed and fed to the 
>>> ecmascript interpreter one at a time, Thus we believe Petr's 
>>> understanding of the specification is correct.
>>>
>>> Chris: Does this resolve the question for you?
>>>
>>> Best regards,
>>>
>>>     RJ
>>>
>>>
>>>
>>> ---
>>> RJ Auburn
>>> CTO, Voxeo Corporation
>>> Chair, Editor and Chair, CCXML, VBWG, W3C
>>>
>>> Come join us at our Voxeo Customer Summit, June 21st – June 23rd at 
>>> the Hard Rock Hotel, register today for your All Access Pass:
>>> http://www.voxeo.com/summits/customer
>>>
>>>
>>>
>>> On Apr 24, 2010, at 4:24 AM, Chris Davis wrote:
>>>
>>>
>>>> Hello again Petr,
>>>>
>>>> I assume your interpreter (OptimTalk) is exposing individual ECMA 
>>>> equivalent lines (except<script>  blocks) from the CCXML document 
>>>> to its ECMA engine
>>>> one line at a time and that is how you manage to pass the test 
>>>> case. Your ECMA engine then has not seen the<var>
>>>> declaration when it begins executing<transition>, allowing you to 
>>>> pass the test case.
>>>>
>>>> When you say:
>>>> "I don't see any statement in the CCXML specification that<var>  
>>>> usage implies declaration of the variable at the begging 
>>>> of<transtion>. "
>>>>
>>>> the closest text seems to be 
>>>> here:http://www.w3.org/TR/2010/CR-ccxml-20100401/#Assign
>>>>
>>>> where the Recommendation says "Variables with transition scope are 
>>>> declared by<var>  and<script>  child elements of<transition>. "
>>>> [ declaration seems to happen separately and prior to execution , 
>>>> like ECMA script itself ]
>>>>
>>>> Then it describes what happens after  declaration:"The child<var>  
>>>> and<script>  elements of<transition>  are initialized in document 
>>>> order when the executable content is executed. "
>>>>
>>>> When you say the following is illegal according to CCXML:
>>>>
>>>> ------------------------
>>>>    <transition event="ccxml.loaded">
>>>>      <assign name="x" expr="3"/>
>>>>      <var name="x"/>
>>>>    </transition>
>>>> -----------------------------
>>>> because of CCXML specification, Section 8.2.1.1:
>>>>
>>>> "It is illegal to make an assignment to a variable that has not been
>>>> explicitly declared using<var>  or a var statement within a<script>."
>>>> ---------------------------
>>>> I must point out that you *did* declare a<var>. It's right there 
>>>> after the<assign>,
>>>> and it goes right into the<transition>  scope during the 
>>>> declaration phase.
>>>>
>>>> Our integration(SPOT) sees<assign>  and turns that into 
>>>> "variablename semicolon" (with a few exceptions, like object/array 
>>>> references)
>>>> which will cause an ECMA error in the cases where there *is 
>>>> no<var>  found in the scope chain*,which is
>>>> my interpretation of the CCXML rule.
>>>>
>>>> Example:
>>>> <assign name="x" expr="1"/>
>>>>
>>>> is *not*
>>>> x = 1; // this would declare/assign x if not found
>>>>
>>>> but instead
>>>> x; // will cause ECMA error if ECMA can't find it
>>>> x=1;
>>>>
>>>> Sure, we can get RJ Auburn's input.
>>>>
>>>> Thanks for the lively discussion.
>>>>
>>>> Regards,
>>>> Chris
>>>>
>>>>
>>>> Petr Kuba wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>>
>>>>>> I don't see how any CCXML implementation can be passing this test 
>>>>>> case.
>>>>>> Is yours passing?
>>>>>>
>>>>> Yes, OptimTalk passes this test smoothly.
>>>>>
>>>>>
>>>>> Your explanation below is correct but it is important to note that 
>>>>> you deal with a script. The following script in OptimTalk would 
>>>>> behave the same way as you describe:
>>>>>
>>>>>     <transition event="ccxml.loaded">
>>>>>       <var name="test"/>
>>>>>       <script>
>>>>>       [![CDATA[
>>>>>         test = id;
>>>>>
>>>>>         var id = 3;
>>>>>       ]]>
>>>>>       </script>
>>>>>     </transition>
>>>>>
>>>>> Here the test value would be 'undefined'.
>>>>>
>>>>> However, the test case deals with<var>  tags. I assume that 
>>>>> the<var>  tag is evaluated according to its position in a CCXML 
>>>>> document. I don't see any statement in the CCXML specification 
>>>>> that<var>  usage implies declaration of the variable at the 
>>>>> begging of<transtion>.
>>>>>
>>>>> I agree with you that it should be clarified whether the CCXML 
>>>>> interpreter behavior here should be the same as for ECMA Script. 
>>>>> This is probably issue for RJ Auburn.
>>>>>
>>>>> Anyway it is also important to consider consequences. For 
>>>>> instance, according to what you've described, this is correct code 
>>>>> in ECMA script:
>>>>>
>>>>>     x = 3;
>>>>>     var x;
>>>>>
>>>>> So if we want to follow the same behavior in CCXML this should be 
>>>>> also correct:
>>>>>
>>>>>     <transition event="ccxml.loaded">
>>>>>       <assign name="x" expr="3"/>
>>>>>       <var name="x"/>
>>>>>     </transition>
>>>>>
>>>>> which probably breaks the following statement in the CCXML 
>>>>> specification, Section 8.2.1.1:
>>>>>
>>>>>   "It is illegal to make an assignment to a variable that has not 
>>>>> been
>>>>>   explicitly declared using<var>  or a var statement within 
>>>>> a<script>."
>>>>>
>>>>> Regards,
>>>>> Petr
>>>>>
>>>>>
>>>> -- 
>>>> Chris Davis
>>>> Interact Incorporated R&D
>>>> 512-502-9969x117
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>> -- 
>> Chris Davis
>> Interact Incorporated R&D
>> 512-502-9969x117
>>
>
>


-- 
Chris Davis
Interact Incorporated R&D
512-502-9969x117

Received on Wednesday, 28 April 2010 15:36:31 UTC