RE: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]

Jacob,
The point of the implementation test suite is to show that there is _no difference_ in behavior between implementations (this refers only to behavior mandated by the spec, of course -implementations are free to do whatever they want in areas not covered by the spec.)  That's the reason for the SCXML group to produce the tests and why the W3C requires that at least two implementations pass the test for each feature defined in the spec.  If we cannot show that two platforms implement the feature the way we have defined it (i.e., by passing our test),  the W3C will require that the feature be removed from the spec (on the grounds that we haven't demonstrated that it can be implemented in an interoperable manner.)

Once the tests are published, we are obviously interested in comments on them, particularly if people think that there are mistakes (i.e., that the test does not correctly capture the behavior defined in the specification.)  But overall, the tests are logically part of the specification and are defined by the same group.

The issue of test harnesses is interesting.  The tests that we provide are actually not written directly in SCXML, but rather in a higher-level language that abstracts away from the underlying data model and certain other details of the language.  To run the tests, you must run the .txml files though an XSLT stylesheet to produce the .scxml file for the actual test.  We provide sample XSLT stylesheets mapping onto the ECMA and XPath data models, but implementations can edit them to fit their own platforms.  (Obviously the tests that we provide for the optional ECMA and XPath data models are written directly in those languages with no need for transformation, but an implementation that uses some other data model can run all the other tests by editing the stylesheet to output their datamodel.)

One of the things that the .txml files abstract away is how success/failure is recorded.  Our default stylesheets just print the result to the screen, but implementations could modify them to write to a DB or record the result in some other way.

On the specific issue of programming languages, however,  SCXML is defined without reference to any implementation language, so in that sense a .scxml file that are the output of the XSLT transformation are  as language- and platform-independent one can get.  Other implementers may be interested in your harness, however.


-          Jim

From: Jacob Beard [mailto:jbeard4@cs.mcgill.ca]
Sent: Tuesday, January 22, 2013 10:27 AM
To: Voice Browser Working Group; www-voice@w3.org
Subject: Re: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]

Hi,

I also think the issue of an SCXML test suite is very important, and I would like to contribute two comments.

First, I think that many existing open source SCXML implementations have written their own test suites. Would it be useful to issue an "RFTC" (Request for Test Cases?) so that these existing implementations can contribute their tests to the W3C for inclusion as part of an official, standard test suite? This would also help to illustrate differences in semantics between existing implementations.

Second, regarding testing infrastructure, an important part of the work I did on SCION [0] was to create a language-independent test harness [1]. This test harness could potentially allow SCXML implementations to be tested in a uniform way, thus making it easier to ensure compliance.

To provide some background: the SCION engine is written in portable ECMAScript, and so it was possible to write bindings to it for various high-level languages and environments, including the JVM, .NET CLR, CPython, as well as various existing JavaScript shell environments such as Rhino and Node.js. There was a need to test each of these language bindings, and so I created an HTTP-based client-server protocol for testing SCXML implementations. This meant that in order to test each language implementation, one simply had to write an HTTP server in the language in which the SCXML engine API is exposed, and implement the server side of the protocol. This made it possible to test each binding uniformly, independent of language or environment, with minimal effort.

This testing harness is relevant to the W3C SCXML standardization effort, because it would allow the W3C to publish a standard test suite, which could then be easily run against any language implementation. It would save specification implementers the trouble of determining how to run W3C's tests and of rolling their own custom test harness for that test suite. I think providing or recommending a testing harness would be an improvement over providing test cases only.

I look forward to your comments. Thanks,

Jake

[0] https://github.com/jbeard4/SCION
[1] https://github.com/jbeard4/scxml-test-framework

On Tue, Jan 22, 2013 at 9:21 AM, Jim Barnett <Jim.Barnett@genesyslab.com<mailto:Jim.Barnett@genesyslab.com>> wrote:
Stefan,
  On the issue of the conformance test, a comprehensive set of tests will be provided as part of the Implementation Report Test Plan when we publish the Candidate Recommendation of SCXML.  Strictly speaking, these are not "conformance tests" since their purpose is not to validate a specific implementation but to prove that the specification can be implemented.  However, as a practical matter, they're a pretty good imitation of a conformance test.  We would be  delighted if you would  run them and submit a report.

I'm not sure when the tests will be published.  If we receive comments that lead us to make substantive changes to the specification, we will have to publish another Last Call working draft.  Once all comments have been dealt with and we are confident that no further changes are necessary, we will publish the Candidate Recommendation, along with the tests.

We will respond to your other comments once we have had a chance to study them.  If you will included "ISSUE-825" in the subject line of any subsequent correspondence, it will help us keep track of thread automatically.

Best,
Jim

-----Original Message-----
From: Voice Browser Working Group Issue Tracker [mailto:sysbot+tracker@w3.org<mailto:sysbot%2Btracker@w3.org>]
Sent: Tuesday, January 22, 2013 9:13 AM
To: w3c-voice-wg@w3.org<mailto:w3c-voice-wg@w3.org>; www-voice@w3.org<mailto:www-voice@w3.org>
Subject: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]

ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]

https://www.w3.org/Voice/Group/track/issues/825

Raised by: James Barnett
On product: SCXML-LC-Comments

from Stefan Radomski [radomski@tk.informatik.tu-darmstadt.de<mailto:radomski@tk.informatik.tu-darmstadt.de>]

Hey there,

[....]

So if you are using or have used SCXML, please take the time to send us your comments.  The more comments we get, the more confident we can be that we are producing a sound and useful standard.

we implemented the SCXML draft (or large parts of it) in C++[1]. We do feature an ecmascript (v8) and an experimental prolog (SWI) data model. The overall state of the implementation is still rather crude and very much undocumented but it builds fine on MacOSX, Linux and every now and then on Windows.

I guess what we missed the most is a set of conformance tests as some ..scxml files with sequences of configurations the interpreter is supposed to go through and internal events to raise. Especially the logic around "filterPreempted" is still somewhat fuzzy to us.

Apart from that, there are some areas in the draft where we could imagine some improvements. Just off the top of my head:
- When you use the "Null" datamodel, you cannot use the <param> element to pass even literal strings to e.g. an invoker as both "expr" and "location" are subject to evaluation by the datamodel. Maybe the obvious "value" attribute with a literal string could be included.
- Sending events via the basichttp ioprocessor should allow for the other party to send events in the response.
- Is there a reason scopes in the datamodel were disallowed so explicitly?
- When evaluating foreach with "item" already defined in the datamodel, is it to be reset after foreach ends?
- When embedding an interpreter, #_parent could be specified to send events to the embedding application.

I am sure we will run into many more problems once interoperability with other interpreters becomes an issue.

Best regards
Stefan

[1] https://github.com/tklab-tud/uscxml

---
FB20 Telecooperation | Darmstadt University of Technology Hochschulstr. 10 | D-64289 Darmstadt Germany | Room S2|02 / A108 Tel +49 (6151) 16-6670<tel:%2B49%20%286151%29%2016-6670> | Fax +49 (6151) 16-3052<tel:%2B49%20%286151%29%2016-3052>