RE: Thoughts on MTOM testing

Paul Downey writes:

> This does raise another use for the test pack: in
> practically divining the lowest common denominator of
> functionality actually supported by a set of
> implementations which are expected to interoperate.

Exactly my point.  The Recommendations themselves intentionally avoid 
going very far in this direction.  They do make clear that you have to use 
SOAP envelopes, that each envelope needs a body, etc., but most other 
things are optional.  Having a large bucket of test cases that can be 
applied >as appropriate< is very useful.  The trick is to avoid backing 
into defining levels of conformance without doing it carefully.  It may be 
that somebody would want to write a specification for one or more sorts of 
general purpose SOAP implementations, and might e.g. require that 
conforming implementations demonstrate an ability to send a range of 
headers, multiple headers, etc.  The SOAP Rec itself doesn't actually 
require that you do much of anything.  It does say that what you choose to 
do must obey certain rules:  you don't have to send any headers, but any 
you send must be properly formed, etc.

--------------------------------------
Noah Mendelsohn 
IBM Corporation
One Rogers Street
Cambridge, MA 02142
1-617-693-4036
--------------------------------------








<paul.downey@bt.com>
07/29/2004 09:41 AM

 
        To:     <noah_mendelsohn@us.ibm.com>, <mgudgin@microsoft.com>
        cc:     <xml-dist-app@w3.org>
        Subject:        RE: Thoughts on MTOM testing


Noah,

thanks, some interesting points. 

we see two key benefits for developing a good comprehensive set of 
test cases for SOAP and WSDL:

1) testing can only improves the overall quality of a specification
as outlined in the W3C QA Handbook. it highlights ambiguities, anomalies 
and finds inconstancies in the specification.

2) testing using real implementations drives forward implementations
improving actual interoperability.

The partial implementation issue you raise WRT the traffic lights does
concern, but i hope hasn't confused me. SOAP includes a good 
mustUnderstand model, so in the case of an implementation using an 
required 
extension or feature, correctly failing on the basis of not understanding 
a 
required header would not be a fail. This does raise another use for the 
test pack: in practically divining the lowest common denominator of 
functionality actually supported by a set of implementations which are 
expected to interoperate. I may choose to simplify or provide an 
alternative 
simple service if i expect to interoperate with the traffic lights as well 
as
higher functioning parties, but i need to know exactly what to simplify.

Please note, our motivation for encouraging as much practical 
interoperability
testing as possible is not to 'finger-point' at vendors. The success of 
Web services depends upon actual interoperability of production software. 

A test pack which may be used to regressively demonstrate practical 
interoperability of implementations can only be of benefit to us all
when selling the Web services vision.

Paul


-----Original Message-----
From: noah_mendelsohn@us.ibm.com [mailto:noah_mendelsohn@us.ibm.com]
Sent: 28 July 2004 22:48
To: Martin Gudgin
Cc: Downey,PS,Paul,XSJ67A C; xml-dist-app@w3.org
Subject: RE: Thoughts on MTOM testing


Paul Downey writes:

> 
> FWIW i'm trying to put together some test cases for the WSD
> test pack and will try to incorporate as many tests as the XMLP WG 
> elect to provide combined with WSDL 2.0 descriptions. 

Paul:  the XMLP group also informally suggested that I make here a point 
that we discussed on our telcon today.

It's easy to get confused about conformance requirements for something 
like SOAP, which is ultimately a wire format.  Based on the SOAP 
Recommendation you can distinguish valid from invalid SOAP messages, and 
you can determine certain things about the correct interpretation (at a 
given node) of those that are valid.

On the other hand, and this often causes confusion, the SOAP 
Recommendation does not have conformance rules for what you might call a 
general purpose SOAP implementation.  Let's imagine that I am building an 
embedded system, for use in a traffic light perhaps, and I know that in 
this particular application no SOAP headers will ever be needed. According 

to the SOAP Rec. it's just fine for me to build software that is incapable 

of sending SOAP headers, as long as the messages I do send conform to the 
Recommendation.  Viewed differently, what value is there in me being 
forced for conformance reasons to build software into my traffic light if 
that software (I.e. software to send headers) will never be executed?  The 

SOAP Recommendation is carefully worded to allow such flexibility. 

Of course, nothing prohibits anyone from building an additional 
specfication to try and standardize the features that might be found in 
more general purpose implementations, but the SOAP recommendation doesn't 
do that.

I mention all this because any discussion of developing test cases for 
SOAP needs to be seen in the above context.  There are indeed many useful 
test cases that you can create;  indeed we've got some at [1] but we 
carefully note that:

"Even though the purpose of the SOAP 1.2 Test Suite is to facilitate the 
creation of interoperable implementations, conformance to the SOAP 1.2 
Test Suite does not imply conformance to the SOAP 1.2 specifications; 
there are mandatory requirements of the specifications that are not tested 

by the suite (as a simple example, SOAP 1.2 requires that every legal 
value of a role name is accepted, and all illegal ones rejected). An 
implementation may be said to be SOAP 1.2 conformant if and only if it it 
satisfies the conformance requirements specified in SOAP 1.2 
specifications. The W3C does not at this time provide for any 
comprehensive means of testing for such conformance.

Similarly, an implementation may conform to the SOAP 1.2 specifications 
even if it does not support all capabilities tested by the SOAP 1.2 Test 
Suite. SOAP 1.2 specifications admits special purpose implementations, 
such as those in dedicated controllers, which may send and receive only a 
very limited suite of messages; the requirement is that whatever is done 
be done correctly. An implementation may conform to the SOAP 1.2 
specifications even if it does not support all capabilities tested by the 
SOAP 1.2 Test Suite. The test suite defines higher level application 
semantics to enable testing and facilitate interoperable implementations. 
It is not necessary for a SOAP processor to support these higher level 
semantics to be SOAP 1.2 compliant."

I'm not sure how this reflects on WSD test cases, but any test cases 
developed for SOAP should be seen in the above context, I think.  Thank 
you.

Noah

[1] http://www.w3.org/TR/2003/REC-soap12-testcollection-20030624/

--------------------------------------
Noah Mendelsohn 
IBM Corporation
One Rogers Street
Cambridge, MA 02142
1-617-693-4036
--------------------------------------

Received on Wednesday, 4 August 2004 17:24:19 UTC