RE: message contents -- was Minutes - WS-Addressing Test Suite TF call 5 Jan 2006

Hi Paul:

> > What I'm after here is a clear and simple way to look at a given 
> > message exchange and know for sure which test was being executed.
> 
> Agreed, for the debugging part and I think we have that in:
> 
>   "fault-test1234 foo bar snark snork"
>   "test1234 - foo bar snark snork"

Sure, though I'd much rather see the test number always in the exact
same place (i.e. "test1234: foo" / "test1234 fault: bar").

> > This is going to make writing validating endpoints easier 
> (otherwise 
> > you need to configure the endpoint before the messages are 
> received -
> 
> by 'validating endpoints' I take it you mean 'an observer':
> http://www.w3.org/2002/ws/addr/testsuite/observer/

No, I mean a validating endpoint.  I'm fine with this "observer" idea,
but what if I want to just confirm myself that the messages look right?
I want an endpoint that receives a message, checks it against the
specified conditions for that particular test, and spits out a "yea" or
a "nay" into a log file.  If I can't know which message I'm receiving by
looking at the contents of the message in a standard way, how can I
possibly do this?  For that matter, how can the observer even do this?

Wouldn't it be nice if people (even W3C) could eventually put up online
endpoints which check conformance using these tests?  I'd like those
endpoints to be able to check individual tests rather than going through
the whole sequence or relying upon a human to know which conditions to
check.  Making it obvious what's going on seems a necessary step if
that's to be possible.

> > so I'm not exactly sure how you can say the tests should 
> work without 
> > being keyed on the test number?), it's going to make 
> debugging easier 
> > (not that we'll need any of that of course :)), and it's 
> going to make 
> > reading the log files easier, not to mention writing up the results.
> 
> We discussed this a couple of times on the TF calls. We have 
> a standard log file format which tags each message with the 
> test case identifier: 
> 
> http://www.w3.org/2002/ws/addr/testsuite/logs/
> 
> We decided to side-step how that taggging is actually done, 
> but it could be added by hand or prior agreement on the order 
> of messages.

Ew.  Again I ask - why make things so hard on us the poor testers?
We're writing the rules for this thing, and it's not as if it's
particularly hard to just put some kind of indication in the message
(again, either in the string or a SOAP header or a separate element,
whatever) of which test this is.

> However I agree that recognising the test case based upon the 
> content of the messages is simpler, at least during our 
> bootstrap / debugging phase, hence the recommended optional mode:
> 
> http://www.w3.org/2002/ws/addr/testsuite/#operations
> 
> > Honestly I don't care if we use a fixed string format or use a SOAP 
> > header, but I feel pretty strongly that it's a mistake to skip this 
> > kind of thing.
> 
> OK, and I think we have that, but we hope you're not reliant 
> upon the test case identifier being present in the message 
> for the 'final run'.

What I'm saying is that I want this non-optional, written into the test
suite, as part of the rules, so that everyone can rely on it and
therefore make our lives much easier.

I am not tied to how this is done.  If we want to have a different
operation for each test instead of overloading two of them, that's fine
too (as long as there are different GEDs).  Relying on something like
the ordering of messages (dropped messages therefore screw the rest of
your run) or external injection of test number (slow and inefficient)
seems much less optimal than a simple in-message solution.  Then you can
run the tests as fast and in whatever order you want.

Let me turn this around - what's so hard about this?  Why *don't* we
want to go there?

--Glen

Received on Monday, 9 January 2006 17:53:31 UTC