Notes of the QA TF meeting 20 May 2004

QA Task Force brief notes:

Present: Arthur, Paul, Allen, Jonathan

Discussed IP of test cases [38].  If non-members submit, we'll ask for a
license.

Need new mailing list for test suite submission and discussion.  Ask
Hugo for one, access restricted to members/those who have submitted the
test case.  Will commence with this after the AB plan is implemented.

Need a test case framework (XML document).  Discussed how WS-I Test
Assertion Documents work.  We will have a Test Assertion Document, which
is a formalization of the spec.  We also need a document which links
test cases to TAD assertions.  Should we link passing test cases to test
assertions?  Do we want dependencies between test assertions?

Framework document should:
- list test cases
- categorize test cases as pass or fail
- link failure cases to test assertions
- message correlation

Arthur has used Zed to describe assertions for the Infoset spec, might
be appropriate for the Test Cases.  Arthur thinks Zed translation will
help improve the clarity of the spec too.  But not many people can use
Zed, so it's unlikely to have much use as a formal deliverable of the
WG.  We spent some time looking at Arthur's Infoset translation to Zed
and were very comfortable with a TAD that looks similar, with links.  We
would add Zed schema names as identifiers as needed into the WSDL specs.

Publish in PDF?  Latec can be output as HTML.  Jonathan will prototype a
Zed schema table in HTML.  We found that most or all of the Zed symbols
have been added to Unicode.  We'll see if they can be displayed in
browsers.

Other than getting the spec to Last Call, there is nothing substantial
blocking progress on creating the Test Assertion Document.

Coverage: [36] At least one failure for each test assertion.  Success
cases would need to cover each named thing in the schema.  We can
compare that to names used in each good test case.  By comparing these,
we can find any holes in our test bucket.  Paul volunteers to work on a
tool to do this analysis.

We want to provide a facility to link up message exchanges with a WSDL.
We could do a similar cross-analysis between a WSDL and a message to
ensure that test coverage is complete.  We can use the WS-I log format
to capture messages.

In general, Arthur's plan is good, we'll proceed.

Also introduced ExIT [37] briefly, will try a prototype and then present
to the WG if appropriate.

[35]
http://lists.w3.org/Archives/Public/www-ws-desc/2004Apr/att-0029/QA_Oper
ational_Checklist.htm
 [36] http://lists.w3.org/Archives/Public/www-ws-desc/2004Apr/0037.html
 [37] http://www.w3.org/2003/12/exit
 [38] http://www.w3.org/2004/Talks/05AC-AB/slide6-0.html

Received on Thursday, 20 May 2004 15:25:07 UTC