RE: Thoughts on MTOM testing

Noah,

thanks, some interesting points. 

we see two key benefits for developing a good comprehensive set of 
test cases for SOAP and WSDL:

1) testing can only improves the overall quality of a specification
as outlined in the W3C QA Handbook. it highlights ambiguities, anomalies 
and finds inconstancies in the specification.

2) testing using real implementations drives forward implementations
improving actual interoperability.

The partial implementation issue you raise WRT the traffic lights does
concern, but i hope hasn't confused me. SOAP includes a good 
mustUnderstand model, so in the case of an implementation using an required 
extension or feature, correctly failing on the basis of not understanding a 
required header would not be a fail. This does raise another use for the 
test pack: in practically divining the lowest common denominator of 
functionality actually supported by a set of implementations which are 
expected to interoperate. I may choose to simplify or provide an alternative 
simple service if i expect to interoperate with the traffic lights as well as
higher functioning parties, but i need to know exactly what to simplify.

Please note, our motivation for encouraging as much practical interoperability
testing as possible is not to 'finger-point' at vendors. The success of 
Web services depends upon actual interoperability of production software. 

A test pack which may be used to regressively demonstrate practical 
interoperability of implementations can only be of benefit to us all
when selling the Web services vision.

Paul


-----Original Message-----
From: noah_mendelsohn@us.ibm.com [mailto:noah_mendelsohn@us.ibm.com]
Sent: 28 July 2004 22:48
To: Martin Gudgin
Cc: Downey,PS,Paul,XSJ67A C; xml-dist-app@w3.org
Subject: RE: Thoughts on MTOM testing


Paul Downey writes:

> 
> FWIW i'm trying to put together some test cases for the WSD
> test pack and will try to incorporate as many tests as the XMLP WG 
> elect to provide combined with WSDL 2.0 descriptions. 

Paul:  the XMLP group also informally suggested that I make here a point 
that we discussed on our telcon today.

It's easy to get confused about conformance requirements for something 
like SOAP, which is ultimately a wire format.  Based on the SOAP 
Recommendation you can distinguish valid from invalid SOAP messages, and 
you can determine certain things about the correct interpretation (at a 
given node) of those that are valid.

On the other hand, and this often causes confusion, the SOAP 
Recommendation does not have conformance rules for what you might call a 
general purpose SOAP implementation.  Let's imagine that I am building an 
embedded system, for use in a traffic light perhaps, and I know that in 
this particular application no SOAP headers will ever be needed. According 
to the SOAP Rec. it's just fine for me to build software that is incapable 
of sending SOAP headers, as long as the messages I do send conform to the 
Recommendation.  Viewed differently, what value is there in me being 
forced for conformance reasons to build software into my traffic light if 
that software (I.e. software to send headers) will never be executed?  The 
SOAP Recommendation is carefully worded to allow such flexibility. 

Of course, nothing prohibits anyone from building an additional 
specfication to try and standardize the features that might be found in 
more general purpose implementations, but the SOAP recommendation doesn't 
do that.

I mention all this because any discussion of developing test cases for 
SOAP needs to be seen in the above context.  There are indeed many useful 
test cases that you can create;  indeed we've got some at [1] but we 
carefully note that:

"Even though the purpose of the SOAP 1.2 Test Suite is to facilitate the 
creation of interoperable implementations, conformance to the SOAP 1.2 
Test Suite does not imply conformance to the SOAP 1.2 specifications; 
there are mandatory requirements of the specifications that are not tested 
by the suite (as a simple example, SOAP 1.2 requires that every legal 
value of a role name is accepted, and all illegal ones rejected). An 
implementation may be said to be SOAP 1.2 conformant if and only if it it 
satisfies the conformance requirements specified in SOAP 1.2 
specifications. The W3C does not at this time provide for any 
comprehensive means of testing for such conformance.

Similarly, an implementation may conform to the SOAP 1.2 specifications 
even if it does not support all capabilities tested by the SOAP 1.2 Test 
Suite. SOAP 1.2 specifications admits special purpose implementations, 
such as those in dedicated controllers, which may send and receive only a 
very limited suite of messages; the requirement is that whatever is done 
be done correctly. An implementation may conform to the SOAP 1.2 
specifications even if it does not support all capabilities tested by the 
SOAP 1.2 Test Suite. The test suite defines higher level application 
semantics to enable testing and facilitate interoperable implementations. 
It is not necessary for a SOAP processor to support these higher level 
semantics to be SOAP 1.2 compliant."

I'm not sure how this reflects on WSD test cases, but any test cases 
developed for SOAP should be seen in the above context, I think.  Thank 
you.

Noah

[1] http://www.w3.org/TR/2003/REC-soap12-testcollection-20030624/

--------------------------------------
Noah Mendelsohn 
IBM Corporation
One Rogers Street
Cambridge, MA 02142
1-617-693-4036
--------------------------------------

Received on Thursday, 29 July 2004 09:41:35 UTC