WSDL Interop progress report

The Interop event held July 5-8th in Toronto marked substantial progress
towards are CR implementation goals.  Present were IBM (Arthur Ryman,
John Kaputin, Lawrence Mandel), Canon (Youenn Fablet), University of
Illinois/Axis2 (Chathura Herath), W3C (Philippe Le Hegaret), and myself.

 

Test suite health:

 

The specs (Core, Adjuncts) contain 226 machine-readable assertions.
Philippe crafted a report [1] to categorize and list the assertions,
correlate them to test cases, and thereby measure our progress towards a
test suite that contains at least one test case violating each
assertion.  In the process we more finely tuned the test metadata
formats.  Our bucket of bad documents grew from 15 to 60.

 

Our bucket of good documents likewise grew, from 35 testcases to 64.
The test coverage report [2] shows that 42% of the constructs are
covered by two or more testcases.  Or if you rather, 51% is covered by
one or more testcases.

 

Summary:

*	Validation test suite completion: 25 of 226 assertions covered:
11% complete.
*	Language coverage completion: 55 of 108 constructs covered by at
least one test case: 51% complete.
*	Language coverage completion: 46 of 108 constructs covered by at
least two test cases: 42% complete.

 

Implementation health:

 

We developed an initial format for reporting validation results [3], and
rolling those results up into a report [4].  The format includes a
facility to check not just for a validation error, but that the
assertion violation has been reported correctly.  Woden is the only
implementation providing preliminary results at this point.

 

The interchange format results [5] continue to grow with the test suite.
Some major bugs were fixed in the Woden implementation, leading to
better results, but much more diagnosis needs to be done, and updated
Canon results need to be obtained.

 

We performed some message exchanges between Apache client/server and
Canon client/server (including self-tests).  We obtained logs of these
exchanges which were apparently successful, but still have to develop a
mechanism to evaluate the results and display the results.

 

Next steps:

*	Develop bad testcases covering remaining 201 assertions.
*	Develop good testcases covering the remaining 50-60 constructs.
*	Improve test coverage report to show percentage complete.
*	Generate and evaluate additional Woden validation results.
*	Generate and evaluate Canon validation results.
*	Complete validation result display.
*	Generate and evaluate additional Woden interchange format
results as new test cases are added.
*	Generate and evaluate additional Canon interchange format
results as new test cases are added.
*	Generate and evaluate additional wsdl-xslt interchange format
results as new test cases are added.
*	Develop mechanism to evaluate message exchanges.
*	Develop dashboard providing convenient display of all
indicators.

 

Still lots to do!  But the implementers present believe that we may be
able to complete the test suite by early September.  We discussed
favorably the possibility of holding another interop event in late
September or early October to attempt to complete the interop work and
enable the specs to move forward to PR shortly after.

 

[1]
http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/assertions-r
eport.html?content-type=text/html;%20charset=utf-8

[2]
http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/test-suite-c
overage-summary.xml?content-type=text/xml

[3]
http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Wode
n/validation-results.xml

[4]
http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/impl
ementations.xml

[5]
http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Over
view.html

 

 [  Jonathan Marsh  ][  jmarsh@microsoft.com
<mailto:jmarsh@microsoft.com>   ][  http://auburnmarshes.spaces.msn.com
]