Suggested Contents of Test Suite

Now that Jonathan has kicked off the QA Plan, it's time for us to get 
serious about the Test Suite (still waiting for the IP policy statement 
though). Here are some initial thoughts on what we can test. Please 
comment.

1. WSDL Documents

WSDL is a language so we can test whether a document belongs to the 
language or not (actually, we need to apply this to a collection of 
documents since we have <import> and <include>). We can encode many, but 
not all of the language rules in XML Schema. The XML Schema is a normative 
part of the WSDL spec. However, there are rules that cannot be expressed 
in XML Schema and we need to record them. We will mine the WSDL spec and 
create a Test Assertion Document (TAD). Each assertion will be given a 
unique, stable, identifier.

Our test suite should consist of two buckets: Good and Bad (I am avoiding 
the use of the term Valid since XML already uses that term to indicate 
compliance with a schema).

The Good bucket consists of documents that are in the language. We should 
aim for adequate test coverage since people will look to this bucket for 
examples of usage of the capabilities of WSDL. I suggest we approach the 
test coverage problem as follows. We should analyse the XML Schema and 
extract a list of every element name, attribute name, and enumerated 
value, and we should aim that each appears in at least one Good test case.

The Bad bucket consists of documents that are not in the language. I 
suggest that our test coverage aims to have each Test Assertion be 
violated by at least one test case in the Bad bucket.

2. Message Instances

The above WSDL document tests address the syntax of the language. The 
semantics of the WSDL language include descriptions of messages. The WSDL 
specification includes binding rules that define the content of concrete 
messages as they are transmitted over some transport protocol, e.g. SOAP 
over HTTP. 

Our test suite should include buckets of Good and Bad message instances 
for each normative binding. Each test case in these buckets consists of a 
triple that contains a Good WSDL document, a message definition for some 
bound operation defined in the document, and a message instance. The test 
case is Good if the WSDL document correctly describes the message 
instance, and is Bad otherwise.

We need to mine the WSDL binding specs and extract a TAD for each binding. 
In the case where the message body is XML, there may be an XML Schema 
which can be used to validate the message, and we should add this to the 
test case.

The Good bucket should contain test cases that illustrate the use of each 
main aspect of the binding. The Bad bucket should contains test cases that 
in total violate each Test Assertion.

3. Interaction Instances

The WSDL spec also defines Message Exchange Patterns (MEP) that define an 
interaction with the Web service. This is part of the semantics of WSDL. A 
test case consists of a triple that contains a Good WSDL document, a bound 
operation defined in the document, and a set of message instances and 
their roles as defined by the MEP of the operation. A test case is Good if 
the message instances conform to the operation, and is Bad otherwise.

We need to mine the WSDL MEP spec and extract a TAD.

Our suite should contain Good test cases for each normative MEP and Bad 
test cases that in total violate each Test Assertion for each MEP.

4. Processor Behaviours

This area needs further clarification in the spec. We need to mine the 
spec and extract a TAD. 

A test case consists of a pair that contains a Good WSDL document, and 
some parameterization of the capabilities of a processor. This 
parameterization remains to be determined, but could consist of a list of 
"understood" namespaces, or an assertion that all namespaces are 
understood.

The Good bucket should contain WSDL documents that illustrate the major 
ways that extensions and be added to documents and be marked with 
wsdl:required. The Bad bucket should contain test cases that in total 
violate each Test Assertion.

Arthur Ryman,
Rational Desktop Tools Development

phone: +1-905-413-3077, TL 969-3077
assistant: +1-905-413-2411, TL 969-2411
fax: +1-905-413-4920, TL 969-4920
mobile: +1-416-939-5063
intranet: http://w3.torolab.ibm.com/DRY6/