Testing: a proposal

This document is intended to satisfy my action from the 27 May 2008 
teleconference.

Coverage:

Test specification implementability.
Test specification ambiguity.
Test implementation conformance.
Test provider conformance.

Assumptions:

The primary product supplied by this portion of the work is a test 
suite, with a well-defined interface which implementors may hook.  The 
term "test suite" means, in this document, a collection of test cases 
(with clear success/failure indicators) and supporting code, tools, 
and documentation.

Testing strategies are designed to localize issues, by testing the 
same message at multiple points.

Minimal code should be supplied.  All test cases should work with the 
same base code.

Support for WSDL cannot be required (it could be the test case 
description language, otherwise).  Test cases must have a formal 
description.

Supplied by the working group:

   Java
     MessageParser
     Context
     (Optionally in Java) WSDL validator

   XSLT
     WSDL 1.1 -> SJMSMDL
     WSDL 2.0 -> SJMSMDL
     (Optionally in XSLT) WSDL validator

   XML
     SJMSMDL schema
     (Optional?) SJMSTCL schema
     Test cases (at least SJMSMDL for each message, possibly SJMSTCL or 
WSDL)

Supplied by the tester:

Message Producer

Message Consumer

JMS Provider

Descriptions:

MessageParser: this Java tool is initialized with a Context data 
object and a Message.  Its role is to generate an XML instance 
document conforming to the SJMSMDL schema.

Context: this Java object contains data not present in a message, as 
defined by the SOAP/JMS binding specification.  For instance, the URI 
supplied to an implementation, timeToLive, JNDI-related properties, 
and so on.

WSDL Validator: since WSDL support is not required of conforming 
implementations, the WSDL validator simply validates (or fails to 
validate) a WSDL containing SOAP/JMS information.  Test cases for 
conformance to the WSDL support functionality verifies that the 
implementation accepts and rejects the same set of WSDLs.

SJMSMDL schema: SOAP over JMS Message Description Language schema.  
Attached (RELAX-NG compact format, initial draft, known incomplete).  
This is an XML micro-format describing the content of a Message and 
(optionally) Context.  The MessageParser code produces this as output. 
  Test cases contain an instance conforming to this schema for each 
message.  Tests pass when the generated instance matches the supplied 
instance.

SJMSTCL schema: SOAP over JMS Test Case Language schema.  Not clear 
whether this is needed.  If so, it would describe the larger context 
of a test case, perhaps as guidelines for comparing the supplied 
versus the generated SJMSMDL instances.

Test cases: infinitely expandable without changing code; add a 
directory containing the necessary bits of XML, and you've got a new 
test case.

Message Producer: the tester is either the working group itself (to 
verify that conformant messages may be produced) or an implementor.  
We don't care what their code looks like; it just has to call 
MessageParser before transmission.

MessageConsumer: the tester is either the working group itself (to 
verify that conformant messages can be consumed) or an implementor.  
We don't care what their code looks like; it just has to call 
MessageParser before consumption.

JMS Provider: for feasibility testing (by the working group), a 
null-ish provider may be supplied.  It would permit creation of 
Message (via standard JMS APIs) by a producer; on send it would hand 
off the message to a consumer (no network transmission).  Vendors of 
JMS may also effectively test provider implementations by verifying 
that no data has been lost between the MessageParser output prior to, 
and subsequent to, transmission.

Test strategies:

Each message in SJMSMDL is marked as "request," "response," or 
"one-way".  Each test case requires that, after production of the 
message, it be handed off (with an optional Context object) to 
MessageParser; the output of MessageParser is compared to the supplied 
message description in the test case, for the outgoing side.  For most 
test case scenarios (except the possible null provider used by the 
working group), the message then passes over the network, through a 
JMS provider implementation.  On the consuming side, the message (and 
an optional Context object) must be handed off to a MessageParser, and 
the output compared with the instance supplied by the test case.

The simplest test cases test producer, provider, and consumer with a 
single message, compared to its definition in the supplied test case 
documents.

More complex test cases test MEP support: one-way (no reply permitted) 
and request-response (reply required).  This is also the area in which 
testing of fault support must be defined.

Test cases must include malformed messages of various sorts, to verify 
that failure happens when it should.

<sjmsmdl.rnc>

Amy!
-- 
Amelia A. Lewis
Senior Architect
TIBCO/Extensibility, Inc.
alewis@tibco.com