W3C home > Mailing lists > Public > www-xkms@w3.org > July 2004

[interop matrix] ideas on how to conduct the tests

From: Jose Kahan <jose.kahan@w3.org>
Date: Thu, 8 Jul 2004 18:57:24 +0200
To: www-xkms@w3.org
Cc: dom@w3.org
Message-ID: <20040708165724.GA20380@inrialpes.fr>

Hi,

While talking with Guillermo today, we found the problem of how we are going
to test multiple clients and servers:

- Should each server be tested by all clients and viceversa?
- What happens when clients and servers only implement parts of the
  spec?
- How can you make sense of the results, claim that your client or
  server are interoperable?

I talked with the QA people and got the following pointers:

- Organize your scenarios into profiles.
- Have a reference client and server for each profile (meaning that they
  implement correctly everything in that scenario)
- Make all the clients tests against the ref. server.
- Make all the server tests against the ref. client.
- Declare in the test suite which is the ref. client and server for each
  set of profiles.
- There can be more than one reference client/server per profile.
- If the client/server works against the ref. version, then it has
  passed the test.

Does this make sense to you?

BTW, I also got a pointer to some guidelines for writing test scenarios[1] and
related info [2].

I asked again to see if we can have someone from the QA-WG attend the
next XKMS teleconf. 

-jose

[1] http://www.w3.org/TR/2004/WD-qaframe-test-20040225/
[2] http://esw.w3.org/topic/QA
Received on Thursday, 8 July 2004 12:57:32 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:31:42 UTC