Re: WSDL Interop progress report

Jonathan,
Its Indiana University/Axis2 not Illinois.
Chathura

On 7/11/06, Jonathan Marsh <jmarsh@microsoft.com> wrote:
>
>
>
>
> The Interop event held July 5-8th in Toronto marked substantial progress
> towards are CR implementation goals.  Present were IBM (Arthur Ryman, John
> Kaputin, Lawrence Mandel), Canon (Youenn Fablet), University of
> Illinois/Axis2 (Chathura Herath), W3C (Philippe Le Hegaret), and myself.
>
>
>
> Test suite health:
>
>
>
> The specs (Core, Adjuncts) contain 226 machine-readable assertions.
> Philippe crafted a report [1] to categorize and list the assertions,
> correlate them to test cases, and thereby measure our progress towards a
> test suite that contains at least one test case violating each assertion.
> In the process we more finely tuned the test metadata formats.  Our bucket
> of bad documents grew from 15 to 60.
>
>
>
> Our bucket of good documents likewise grew, from 35 testcases to 64.  The
> test coverage report [2] shows that 42% of the constructs are covered by two
> or more testcases.  Or if you rather, 51% is covered by one or more
> testcases.
>
>
>
> Summary:
>
> Validation test suite completion: 25 of 226 assertions covered: 11%
> complete.
> Language coverage completion: 55 of 108 constructs covered by at least one
> test case: 51% complete.
> Language coverage completion: 46 of 108 constructs covered by at least two
> test cases: 42% complete.
>
>
>
> Implementation health:
>
>
>
> We developed an initial format for reporting validation results [3], and
> rolling those results up into a report [4].  The format includes a facility
> to check not just for a validation error, but that the assertion violation
> has been reported correctly.  Woden is the only implementation providing
> preliminary results at this point.
>
>
>
> The interchange format results [5] continue to grow with the test suite.
> Some major bugs were fixed in the Woden implementation, leading to better
> results, but much more diagnosis needs to be done, and updated Canon results
> need to be obtained.
>
>
>
> We performed some message exchanges between Apache client/server and Canon
> client/server (including self-tests).  We obtained logs of these exchanges
> which were apparently successful, but still have to develop a mechanism to
> evaluate the results and display the results.
>
>
>
> Next steps:
>
> Develop bad testcases covering remaining 201 assertions.
> Develop good testcases covering the remaining 50-60 constructs.
> Improve test coverage report to show percentage complete.
> Generate and evaluate additional Woden validation results.
> Generate and evaluate Canon validation results.
> Complete validation result display.
> Generate and evaluate additional Woden interchange format results as new
> test cases are added.
> Generate and evaluate additional Canon interchange format results as new
> test cases are added.
> Generate and evaluate additional wsdl-xslt interchange format results as new
> test cases are added.
> Develop mechanism to evaluate message exchanges.
> Develop dashboard providing convenient display of all indicators.
>
>
>
> Still lots to do!  But the implementers present believe that we may be able
> to complete the test suite by early September.  We discussed favorably the
> possibility of holding another interop event in late September or early
> October to attempt to complete the interop work and enable the specs to move
> forward to PR shortly after.
>
>
>
> [1]
> http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/assertions-report.html?content-type=text/html;%20charset=utf-8
>
> [2]
> http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/test-suite-coverage-summary.xml?content-type=text/xml
>
> [3]
> http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Woden/validation-results.xml
>
> [4]
> http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/implementations.xml
>
> [5]
> http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Overview.html
>
>
>
>  [  Jonathan Marsh  ][  jmarsh@microsoft.com  ][
> http://auburnmarshes.spaces.msn.com  ]
>
>


-- 
Chathura Herath
http://people.apache.org/~chathura/
http://chathurah.blogspot.com/

Received on Wednesday, 12 July 2006 05:06:49 UTC