Status and some questions about the XKMS CR test suite

Hello folks,

I had contacted you previously[1] on behalf of the XKMS WG. I'm not
sure if I should send this message to the -wg or the -ig. Please
feel to forward it to the other list if I got it wrong this time.

--- some context

We have made some progress, but we are stumbling in some questions.
We would also like to know if there are any tools available that 
we could reuse or readapt.

What we have now is the following:

 - a common set of public keys[2]	
 - few contributed sample XKMS messages[2], which are
   generated with the above keys
 - draft interoperability matrix[3]
 - a fair number of developers and the beginning of interoperability
   testing.

We still haven't done so, but we'll request that the contributed
messages conform to the XKMS XML schema.

The things we are starting to do (or would like to do) now are:

1. Define a set of scenarios (what you send, what you expect to
   get back, what you're testing)

2. Map the concepts of the spec to the scenarios
   --> so that we can know what properties are handled by
       each message and if the set of messages has succesfully
       covered the spec.

3. Let developers report compliancy with respect to the test
   messages.

4. Fill the interoperability matrix in function of the reports.

This is something similar to what was done for the SOAP test suite[4].

-- some questions

Q1: In XKMS, some of the protocol features are optional. We don't
know if its usual in the context of these tests to  have
a previous agreement on which of these features the clients and
servers must support. 

For example, a client can request a server to translate a public key 
certificate from one format to another and it may return a success or 
a failure. This is implementation dependent.  Should we have a previous 
agreement on which public key certificate concertion operations 
we need to support during interop?

Q2: Do you have some tools that can provide the infrastructure for
   building all of this interoperability things? 

   I was thinking about using RDF for mapping the concepts to the
   scenarios and the implementation reports and WBS for having
   the implementors report which scenarios they support and make
   it easier to update the scenarios and reports. And writing
   a script to generate the implementation matrix from the RDF
   data.

   However, if you have some tools available, we could migrate
   to them before re-inventing the wheel again.

Q3: Could it be possible that someone from your group with experience
  in this kind of test suites join us in our next teleconf? It could
  help target our work easier. Our teleconfs happen one tuesday out of
  two  from 8:30am to 9:30 USA +Pacific Time.

  It could be interesting and it could help us kick-boost our efforts
  in the right direction.


-----

Many thanks in advance for your time,

-jose

[1] http://lists.w3.org/Archives/Public/www-qa-wg/2004Apr/0024.html
[2] http://www.w3.org/2001/XKMS/Drafts/test-suite/
[3] http://www.w3.org/2001/XKMS/Drafts/interop-20040621-3.html
[4] http://www.w3.org/TR/soap12-testcollection/
[5] http://lists.w3.org/Archives/Public/www-xkms/

Received on Friday, 25 June 2004 17:59:37 UTC