Re: testing error cases

There should be no problem getting tests containing
bad input values (e.g. a bad secret) since the messages
are correctly formatted.

Messages failing integrity checks are ok too since you
just take the correct message and flip a bit (or whatever).

Weird request messages (e.g. messed up SOAP, v.v.large
messages) do have to be produced by hand but can be sent
to servers easily enough.

Testing clients handling of weird response messages is
also ok for hand-coded responses (e.g. just configure
the http server to hand back the relevant files), so
long as the weird response doesn't have to reflect the
values from the request.

So I guess we can create some test cases that are easy
enough to test and I'd be keen that folks do do that,
in whatever spare time they have.

Why spare time? Well, there is no formal requirement that
we produce such tests, since our job here is to validate
the xkms spec, and not the implementations. So all such
error cases are really just nice-to-have, right?

Stephen.

PS: The "revoke non-existent cert" case is a bad example.
xkms doesn't "see into" certs and therefore has no idea
whether they exist or not (you have to know about the
issuer and serialNumber ASN.1 constructs for that). We
revoke bindings. Even if a cert does exist, a binding
may not. Even if an xkms revoke works, the relevant
certificate may not have been revoked (e.g. if the xkms
responder is not a CA and never tries to pretend to
be one).

Jose Kahan wrote:

> Hi!
> 
> As part of the test suite, we should try to test not only the succesful
> behaviour, but the errors too.
> 
> For example, what happens when you revoke a certificate that doesn't
> exist, what happens when you use the wrong shared secret, what happens
> when you use the status request and you don't get a sucess reply...
> 
> Guillermo pointed out to me today that it may be difficult to do those
> tests, as in some cases the clients or the servers need to be able to send 
> wrong messages. What we want to test in these cases is what happens 
> when either the client or server receive an invalid message.
> 
> It seems that the SOAP people used fake clients or servers to test
> the error cases. I have no idea how flexible are the current
> implementations.
> 
> Any ideas on how to proceed here are welcome.
> 
> -jose
>

Received on Tuesday, 19 October 2004 16:30:25 UTC