This document contains details of the review of the SOAP Version 1.2 Last Call Working Drafts
against the Quality Assurance specification guidelines
The SOAP Version 1.2 Working Drafts consist of four documents: the "Primer," "Messaging Framework," "Adjuncts," and "Assertions and Test Collection". Overall the testability of the specification is satisfactory, especially with the help of the last document "Assertions and test Collection". The review below consists of analysis of each of the four SOAP 1.2 documents against the QA WG Specification Guidelines checkpoints. In addition, here is the summary of issues found during the review.
For some of the multiple-choice assertions, it is not explicitly defined whether the choice must be consistent by the SOAP Node or not.
For example, in the section 2.4, assertion regarding mustUnderstand SOAP headers that allows to either understand the Header or generate a Fault message. It is not clear under which circumstances the behavior of the SOAP Node MUST remain consistent.
Guideline 1. Define user scenarios.
User scenarios are gathered and explained in details in the "Primer" document, section 2,3 and 4. In addition test cases in the "Assertions and Test Collection" document may serve as a collection of more atomic user scenarios for each assertion in the specification.
1.1 Define the scope of the specification. [Priority 1]
Not Satisfied. The scope of the specification is indirectly explained in the Primer and also in the Introduction to the Part 1. But there is no dedicated section that would explain what is in scope and what is explicitly left out of scope of the specification
1.2 Include Use Cases. [Priority 2]
Satisfied. Use cases are included and explained in details in the Primer, section 2,3 and 4.
1.3 Include examples. [Priority 1]
Satisfied. The SOAP 1.2 specification is rich on examples. In addition a systematic coverage of the assertions with examples may be obtained from the "Assertions and Test Collection" document.
1.4 Include an interpretation section. [Priority 2]
Satisfied. The SOAP Version 1.2 specification contains a Primer with interpretation of the specification applied to the user scenarios.
Guideline 2. Identify what needs to conform and how.
Although the SOAP 1.2 specification describes in great details what SOAP Processor must, should and may conform using the keywords from the RFC2119 and mentions the use of these keywords explicitly in Notational Conventions of the Part 1 and Part 2, there is no dedicated conformance section, which makes it harder for a tester to define the conformance criteria for a SOAP 1.2 processor.
2.1 Identify all classes of product. [Priority 1]
Not Satisfied. SOAP Version 1.2 specification defines only one class of product, SOAP Processor. But it never defines what can be called a "SOAP Processor".
2.2 For each class of product, define the conformance requirements. [Priority 1]
Satisfied. The conformance requirements are defined for each assertion using RFC2119 keywords. But there is no dedicated Conformance section.
2.3 Indicate minimal support requirements. [Priority 3]
Satisfied. The minimal support requirements (MUST assertions in the Part I) are indicated by the claim to use RFC2119 keywords and by the claim that any of the adjuncts defined in the Part 2 is optional. But there is no dedicated Conformance section that would define minimal support requirements.
2.4 Identify which of the categories of object are specified in the document as a whole. [Priority 3]
Satisfied. The SOAP 1.2 specifications defines SOAP 1.2 protocol as it is clearly stated in the introduction to the Part 1 and in the Primer.
Guideline 3. Address the use of profiles to divide the specification.
The SOAP 1.2 specification profiles are SOAP Messaging Framework (Part 1) plus any of the adjuncts (Part 2), e.g. SOAP Messaging Framework + SOAP Data Model+ SOAP Encoding, or SOAP Messaging Framework + SOAP RPC , or SOAP Messaging Framework + SOAP HTTP binding.
Although the each possible profile is described in various places of the Part 2, it would be useful to gather them in a separate section (Conformance section)
3.1 Choose whether or not to have profiles. [Priority 1]
Satisfied. SOAP 1.2 specification chooses to use profiles in the form of "Messaging Framework + Adjunct". It defines an open set of profiles, e.g. one may still conform to the specification by using SOAP 1.2 Messaging framework coupled with his own protocol binding, not described in the Part 2.
3.2 If profiles are chosen, ensure that a table of contents entry is generated. [Priority 1]
Satisfied. The specification does not use the term "profiles", but uses the terms Framework and adjuncts. Those are clearly identified in the Part 1 and Part 2 tables of content.
3.3 If profiles are chosen, indicate whether or not their use is mandatory for conformance. [Priority 1]
Satisfied. The way SOAP 1.2 specification is organized, Messaging Framework is mandatory (with respect to RFC2119 keywords) and adjuncts described in the Part 2 are optional.
3.4 If profiles are chosen, define any minimal requirements. [Priority 2]
3.5 If profiles are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]
Satisfied. The profiles of the SOAP 1.2 specification are build from the Messaging Framework and various combinations of adjuncts. There are no modules or levels as defined in the QA Framework:Spec Guidelines. Discretionary behaviors do exists in the specification, but are locally independently defined (within the Messaging Framework and each of the adjuncts).
Guideline 4. Address the use of modules to divide the specification.
The SOAP 1.2 specification does not use modules as defined in the QA Framework Specification Guidelines.
4.1 Choose whether or not to have modules. [Priority 1]
Satisfied. Modules are not used.
4.2 If modules are chosen, ensure that a table of contents entry is generated. [Priority 1]
Satisfied.
4.3 If modules are chosen, indicate any mandatory conditions or constraints on their usage. [Priority 1]
Satisfied.
4.4 If modules are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]
Satisfied.
Guideline 5. Specify conformance policy.
The only places in the SOAP 1.2 specification that define conformance policy are
It would be beneficiary for the testability to include a Conformance section and gather there general conformance requirements, definition of the specification's subject (SOAP Processor), claims mentioned above.
5.1 Make it clear where there are universal requirements for minimum functionality. [Priority 1]
Satisfied. It is explicitly stated that Messaging Framework (Part I) contains minimum requirements for SOAP Processor (with respect to RFC2119 keywords).
5.2 Make it clear when conformance requirements are strict. [Priority 1]
Satisfied. The keywords of the RFC2119 are used in the specification to define assertions. MUST and SHALL assertions are strict. It could be reiterated in the missing separate Conformance section.
5.3 Make it clear where requirements stop and product-specific extra features begin. [Priority 1]
Satisfied. The boundary between obligatory Messaging Framework (Part I) and optional adjuncts (Part II) is clearly defined.
5.4 The conformance clause should contain or refer to the conformance policy. [Priority 1]
Not Satisfied. There is no conformance clause
5.5 If special conformance terms are used, include a definition in the specification. [Priority 1]
Satisfied. No special conformance terms are used. The RFC2119 keywords are used. Their definition is referenced.
Guideline 6. Clarify the relation between deprecated features and conformance.
There are no deprecated features declared in the SOAP 1.2 specification.
6.1 Identify and clearly indicate each deprecated feature. [Priority 1]
Satisfied.N/A
6.2 For each class of product, specify the level of support required for each deprecated feature and the conformance consequences of the deprecation. [Priority 1]
Satisfied.N/A
6.3 Include an explanation for the deprecation. [Priority 3]
Satisfied.N/A
6.4 Include examples to illustrate how to avoid using deprecated features. [Priority 3]
Satisfied.N/A
6.5 Generate a table of contents entry. [Priority 2]
Satisfied.N/A
Guideline 7. Address the use of levels to divide the specification.
SOAP 1.2 uses combinatorial model of profiles that consist of the Messaging Framework (Part 1) and a combination of adjuncts (Part II). The SOAP 1.2 specification does not use Levels as defined in the QA Framework:Specification Guidelines.
7.1 Choose whether or not to have levels. [Priority 1]
Satisfied.Not used
7.2 If levels are chosen, ensure that a table of contents entry is generated. [Priority 1]
Satisfied.N/A
7.3 If levels are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]
Satisfied.N/A
Guideline 8. Define discretionary behaviors.
The SOAP 1.2 specifications defines several states where SOAP Processor MUST perform one of the several allowed actions. For example a SOAP Processor (Node) receiving a MustUnderstand header targeted at it, MUST either process it or return a fault message.
8.1 Explicitly state the cases and conditions where discretion is allowed and/or expected. [Priority 2]
Satisfied. In all observed cases allowed options are explicitly stated.
8.2 Indicate implementation dependencies and where applicable address, allowable differences between implementations. [Priority 1]
Satisfied. Have not noticed any behavior that is not defined, but not explicitly left at the implementation discretion.
8.3 Describe alternative approaches and the conditions under which an implementation is considered to be conforming. [Priority 1]
Satisfied. Have not found behaviors in the specification for which this is applicable
8.4 Include a statement regarding consistent handling of a discretionary item within an implementation. [Priority 2]
Not Satisfied. Some of the behaviors that allow for several choices do not specify that the result must be consistent or not, for example:
2.4...
Therefore, for every mandatory SOAP header block targeted to a node, that node MUST either process the header block or not process the SOAP message at all, and instead generate a fault (see 2.6 Processing SOAP Messages and 5.4 SOAP Fault).....
This checkpoint in the QA Specification Guideline needs more work in my mind, since in this case the reaction of the SOAP Node can be indeed inconsistent depending on the time of the day, other headers, etc. But those circumstances under which the behavior is consistent, should be defined in the spec.
8.5 Generate a table of contents entry. [Priority 2]
Satisfied.N/A. Discretionary behaviors are localy defined and do not require a separate TOC entry.
Guideline 9. Allow extensions or NOT!
SOAP 1.2 specification explicitly allows extensions (processing extensibilities and bindings) in the Part 1, sections 2,3,4. and specifies the rules for those extensions to follow.
9.1 If extensions are disallowed, explicitly state it. [Priority 3]
Satisfied.N/A.Extensions are allowed, this is explicitly stated.
9.2 If extensions are allowed, explicitly state it. [Priority 1]
Satisfied. See Part 1, Section 3
9.3 If extensions are allowed, make it clear that the extensions do not negate support for required functionality. [Priority 1]
Satisfied. Rules of the sections 2 and 4 for extensions make this clear.
9.4 If extensions are allowed, use a standard mechanism to define the extension. [Priority 3]
Satisfied.Section 2 and 4 define a "standard" rules for processing and binding extensions to follow.
9.5 If extensions are allowed, register or publish them. [Priority 3]
Satisfied. Specification (Part I) recommends to publish extensions as a separate specifications. In fact, Part II contains extensions that follow the rules of the section 2 and 4.
9.6 If extensions are allowed, require that implementations include a way to operate without the extension. [Priority 3]
Satisfied. The Specification Guidelines document needs more work here. Messaging Framework describes how to react if the SOAP Processor(Node) does not understand a processing extension (described by a MustUnderstand SOAP header). But while Part I does not require SOAP 1.2 implementation to use any particular underlying protocol/binding, a SOAP 1.2 implementation has to use some.
9.7 Generate a table of contents entry. [Priority 2]
Satisfied. See section 2,3 and 4 of the Part I.
Guideline 10. Provide a conformance clause.
The SOAP 1.2 specification does not provide a conformance clause, and spread conformance requirements across the text.
10.1 Include a conformance clause. [Priority 1]
Not Satisfied.
10.2 Create a separate conformance section. [Priority 2]
Not Satisfied. Not clear what is the difference between this and the previous checkpoint
10.3 Generate a conformance clause entry in the table of contents. [Priority 2]
Not Satisfied.
Guideline 11. Specify how to make conformance claims.
This is not included in the SOAP 1.2 specification.
11.1 Identify and define all conformance levels or designations. [Priority 1]
Satisfied. The SOAP 1.2 specification uses RFC2119 keywords to locally level each particular requirement. The SOAP 1.2 does not use conformance levels, it uses profiles that consist of Messaging Framework and a combination of adjuncts. So in terms of QA Specification Guidelines, SOAP 1.2 specification has 1 level of conformance.
11.2 Provide specific wording of the claim. [Priority 3]
Not Satisfied.
11.3 Provide a conformance disclaimer. [Priority 3]
Not Satisfied.
11.4 Impose no restrictions about who can make a claim or where claims can be published. [Priority 1]
Satisfied. No restrictions imposed.
11.5 Generate a table of contents entry. [Priority 2]
Not Satisfied.
Guideline 12. Publish an Implementation Conformance Statement proforma.
The SOAP 1.2 does not publish an Implementation Conformance Statement proforma.
12.1 Include an Implementation Conformance Statement proforma as part of the specification. [Priority 3]
Not Satisfied.
12.2 Require the ICS be completed as part of the conformance claim. [Priority 3]
Not Satisfied.
Guideline 13. Support general document conformance conventions.
The SOAP 1.2 specification uses RFC 2119 keywords.
13.1 Use conformance key words. [Priority 1]
Satisfied.
13.2 Distinguish normative and informative text. [Priority 2]
Satisfied. Part 0 contains informative text. Part I and Part II contain normative text except those sections that are marked as Non-Normative. That could be explicitly stated in the conformance section.
13.3 Follow Web Accessibility Initiative and Internationalization Guidelines. [Priority 1]
Not verified. I assume this will be verified by WAI or other appropriate W3C group.
13.4 Use the same words to express the same ideas. [Priority 1]
Satisfied. I find terminology to be consistent throughout the document. The QA spec guidelines need to word this checkpoint better.
Guideline 14. Use granular grammars to author the specification.
This is not verified, but I believe SOAP specification uses xml-spec.
14.1 Use W3C endorsed grammar where applicable. [Priority 1]
Not verified. This is not verified, but I believe SOAP specification uses xml-spec.
14.2 Specify intended behavior in the specification using markup. [Priority 1]
14.3 Supply prose description of intended behavior together with each test assertion. [Priority 1]
Satisfied.
Guideline 15. Include test assertions.
XML Protocol WG provides a list of test assertions together with tests in a separate document "Assertions and Test Collection".
15.1 Supply test assertions in the markup of the specification, if applicable using a set of predefined tags used in the specification markup language. [Priority 1]
Satisfied. XML Protocol WG provides a list of test assertions together with tests in a separate document "Assertions and Test Collection".
15.2 Tag test assertions according to the above. [Priority 1]
Not Satisfied.
[Optional: opinions, observations, assessment, judgments, etc, about the subject of your review, about the review process, the Guidelines document, etc.]