Re: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]

Hi Stefan,

The SCXML Test Framework project has two main purposes:

   1. To collect test cases to run against SCXML implementations.
   2. To implement a generic test runner client for a client-server, HTTP-
   and JSON-based SCXML testing protocol.

Contributing (1) would be difficult, because as you've noted, SCION
semantics are explicitly divergent from the semantics described in the
SCXML draft spec, and it would be difficult to isolate test cases that
would not apply to the SCXML spec. I haven't been working on reuniting
SCION and SCXML semantics, but now might be the time to do so. I'll start
this discussion in a new thread.

However, (2) could be contributed, as it's independent of any particular
test suite or implementation.

Best,

Jake

On Tue, Jan 22, 2013 at 11:35 AM, Stefan Radomski <
radomski@tk.informatik.tu-darmstadt.de> wrote:

>  Hey Jake,
>
>  we will have a look at your test harness, but reading[1] I can't help
> but wonder whether we will have a SCION- or a SCXML-interpreter once we
> pass all tests in the harness. Is there still an ongoing effort to reunite
> both semantics again? In any case, could you identify the SCION semantics
> related tests more explicitly until this issue is resolved?
>
>  Regards
> Stefan
>
>  [1] https://github.com/jbeard4/SCION/wiki/SCION-vs.-SCXML-Comparison
>
>  On Jan 22, 2013, at 4:26 PM, Jacob Beard <jbeard4@cs.mcgill.ca>
>  wrote:
>
>  Hi,
>
>  I also think the issue of an SCXML test suite is very important, and I
> would like to contribute two comments.
>
>  First, I think that many existing open source SCXML implementations have
> written their own test suites. Would it be useful to issue an "RFTC"
> (Request for Test Cases?) so that these existing implementations can
> contribute their tests to the W3C for inclusion as part of an official,
> standard test suite? This would also help to illustrate differences in
> semantics between existing implementations.
>
>  Second, regarding testing infrastructure, an important part of the work
> I did on SCION [0] was to create a language-independent test harness
> [1]. This test harness could potentially allow SCXML implementations to be
> tested in a uniform way, thus making it easier to ensure compliance.
>
>  To provide some background: the SCION engine is written in portable
> ECMAScript, and so it was possible to write bindings to it for various
> high-level languages and environments, including the JVM, .NET CLR,
> CPython, as well as various existing JavaScript shell environments such as
> Rhino and Node.js. There was a need to test each of these language
> bindings, and so I created an HTTP-based client-server protocol for testing
> SCXML implementations. This meant that in order to test each language
> implementation, one simply had to write an HTTP server in the language in
> which the SCXML engine API is exposed, and implement the server side of the
> protocol. This made it possible to test each binding uniformly, independent
> of language or environment, with minimal effort.
>
>  This testing harness is relevant to the W3C SCXML standardization
> effort, because it would allow the W3C to publish a standard test suite,
> which could then be easily run against any language implementation. It
> would save specification implementers the trouble of determining how to run
> W3C's tests and of rolling their own custom test harness for that test
> suite. I think providing or recommending a testing harness would be an
> improvement over providing test cases only.
>
>  I look forward to your comments. Thanks,
>
>  Jake
>
>  [0] https://github.com/jbeard4/SCION
> [1] https://github.com/jbeard4/scxml-test-framework
>
> On Tue, Jan 22, 2013 at 9:21 AM, Jim Barnett <Jim.Barnett@genesyslab.com>wrote:
>
>> Stefan,
>>   On the issue of the conformance test, a comprehensive set of tests will
>> be provided as part of the Implementation Report Test Plan when we publish
>> the Candidate Recommendation of SCXML.  Strictly speaking, these are not
>> "conformance tests" since their purpose is not to validate a specific
>> implementation but to prove that the specification can be implemented.
>>  However, as a practical matter, they're a pretty good imitation of a
>> conformance test.  We would be  delighted if you would  run them and submit
>> a report.
>>
>> I'm not sure when the tests will be published.  If we receive comments
>> that lead us to make substantive changes to the specification, we will have
>> to publish another Last Call working draft.  Once all comments have been
>> dealt with and we are confident that no further changes are necessary, we
>> will publish the Candidate Recommendation, along with the tests.
>>
>> We will respond to your other comments once we have had a chance to study
>> them.  If you will included "ISSUE-825" in the subject line of any
>> subsequent correspondence, it will help us keep track of thread
>> automatically.
>>
>> Best,
>> Jim
>>
>> -----Original Message-----
>> From: Voice Browser Working Group Issue Tracker [mailto:
>> sysbot+tracker@w3.org]
>> Sent: Tuesday, January 22, 2013 9:13 AM
>> To: w3c-voice-wg@w3.org; www-voice@w3.org
>> Subject: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]
>>
>> ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]
>>
>> https://www.w3.org/Voice/Group/track/issues/825
>>
>> Raised by: James Barnett
>> On product: SCXML-LC-Comments
>>
>> from Stefan Radomski [radomski@tk.informatik.tu-darmstadt.de]
>>
>> Hey there,
>>
>> [....]
>>
>> So if you are using or have used SCXML, please take the time to send us
>> your comments.  The more comments we get, the more confident we can be that
>> we are producing a sound and useful standard.
>>
>> we implemented the SCXML draft (or large parts of it) in C++[1]. We do
>> feature an ecmascript (v8) and an experimental prolog (SWI) data model. The
>> overall state of the implementation is still rather crude and very much
>> undocumented but it builds fine on MacOSX, Linux and every now and then on
>> Windows.
>>
>> I guess what we missed the most is a set of conformance tests as some
>> ..scxml files with sequences of configurations the interpreter is supposed
>> to go through and internal events to raise. Especially the logic around
>> "filterPreempted" is still somewhat fuzzy to us.
>>
>> Apart from that, there are some areas in the draft where we could imagine
>> some improvements. Just off the top of my head:
>> - When you use the "Null" datamodel, you cannot use the <param> element
>> to pass even literal strings to e.g. an invoker as both "expr" and
>> "location" are subject to evaluation by the datamodel. Maybe the obvious
>> "value" attribute with a literal string could be included.
>> - Sending events via the basichttp ioprocessor should allow for the other
>> party to send events in the response.
>> - Is there a reason scopes in the datamodel were disallowed so explicitly?
>> - When evaluating foreach with "item" already defined in the datamodel,
>> is it to be reset after foreach ends?
>> - When embedding an interpreter, #_parent could be specified to send
>> events to the embedding application.
>>
>> I am sure we will run into many more problems once interoperability with
>> other interpreters becomes an issue.
>>
>> Best regards
>> Stefan
>>
>> [1] https://github.com/tklab-tud/uscxml
>>
>> ---
>> FB20 Telecooperation | Darmstadt University of Technology Hochschulstr.
>> 10 | D-64289 Darmstadt Germany | Room S2|02 / A108 Tel +49 (6151) 16-6670| Fax +49
>> (6151) 16-3052
>>
>>
>>
>>
>>
>
>