RE: Random comments on testing

David

*  of course it remains questionable how we can ever test abstract 
   'core' features without looking 'under the kimono' (TM). 
   All we can do is look at the bits that hit the wire in SOAP and 
   WSDL form.

*  currently we have two 'operations':
   - notify (sends some text, node 'A' to node 'B')
   - echo (send some text, node 'A' to node 'B', same text comes back to node 'A')

   I'll add some documentation on the semantics of these,
   along with proto WSDL 1.1 and 2.0 documents during the next week.

* The test suite has been constructed from the perspective of an
  independent 'observer' of the messages travelling on the wire.

  Either the message exchange described occurs and the test case  
  was demonstrated to the satisfaction of the 'observer', or it 
  doesn't, and the test case wasn't demonstrated. 

  I'm not sure I understand the value of bucketing timeouts 
  differently to invalid messages.

How we actually setup and run implemetations to exhibit these messages  
for CR has yet to be discussed. 


Paul


-----Original Message-----
From: public-ws-addressing-request@w3.org on behalf of David Hull
Sent: Mon 10/24/2005 8:58 PM
To: public-ws-addressing@w3.org
Subject: Random comments on testing
 
I'm still looking to go through all the test docs from top to bottom,
together with the spec.  Unfortunately, I've been fighting a cold
lately, rendering my analysis and prose even more suspect than usual. 
So instead of a lengthy analysis, which would likely come out garbled,
here are a couple of random points:

    * It's hard to test rules concerning abstract models directly.  We
      get at them indirectly, for instance in test 1234, which will only
      work if both the defaulting of [fault endpoint] and the handling
      of [reference properties] are working properly.  This is a good
      test, but I'm wondering if we might want to define a "echo MAPs"
      request-reply operation.  The request is anything, the reply is an
      XML representation of the MAPs, as per the rules in section 3,
      with all defaulting applied.  On the plus side, this would give
      nice, unambiguous results.  On the minus side, it's not something
      that implementations would normally be required to do.  Further,
      it's not strictly necessary for an implementation to build any
      explicit representation of the MAPs at all, as long as it does the
      right thing.
    * Looking at the test cases document
      <http://www.w3.org/2002/ws/addr/testsuite/testcases/>, it's not
      always clear what's supposed to happen.  A test is a repeatable
      procedure and an expected result.  We need to be explicit about
      the result, as well as providing enough information to reproduce
      the procedure.
    * In testing distributed messaging, it's useful to distinguish four
      outcomes:
          o Success:  Everything that was expected to happen happened
            and nothing else happened (at least not where we were looking).
          o Failure:  At least one thing that definitely should not have
            happen happened.
          o Timeout: At least one thing that should have happened didn't
            happen, but might have happened if we had kept waiting.
          o Other failure:  Something identifiable went wrong with the
            test system (e.g., the test driver NPE'd, or the network
            crashed).
    * Timeout is basically a failure, but it's worth calling out because
      it can indicate either a real failure (the reply never arrived at
      the [reply endpoint] because it was never sent) or another failure
      (the reply was sent, but the network hiccuped).  This is more of a
      problem with true one-way transports.  If the server tries to send
      an asynchronous reply via HTTP POST, it can tell us that the POST
      failed.

Received on Wednesday, 26 October 2005 22:17:23 UTC