Re: Interop testing

Mark, Paul and Anish
Thanks for all the comments, they are very helpful - I realise that this 
is early days but it's good to get an idea of what's involved.
I suspect that concalls would be very useful - I think Paul suggested (at 
the f2f) a subcommittee focussed specifically on interop testing (in order 
to focus on the test bucket) prior to the actual testing phase also? 
Katy



Anish Karmarkar <Anish.Karmarkar@oracle.com> 
Sent by: public-ws-addressing-request@w3.org
09/08/2005 19:19

To
Mark Nottingham <mark.nottingham@bea.com>
cc
Katy Warr/UK/IBM@IBMGB, paul.downey@bt.com, public-ws-addressing@w3.org
Subject
Re: Interop testing







Another thing that has helped in the past (for XMLP) was -- having 
concalls (non-official, non-WG) for implementers. This was useful not in 
the initial phase of the interop effort but somewhere in the middle of 
the effort when implementers had specific issues/problems with specific 
implementations or with specific tests (which in certain cases, resulted 
in changes to the tests). These issues/problems were easier to discuss 
on a concall rather than a long email thread.

I would suggest having such concalls, if the implementers think it would 
be useful.

-Anish
--

Mark Nottingham wrote:
> 
> Hi Katy,
> 
> On 05/08/2005, at 3:34 AM, Katy Warr wrote:
> 
>> 1. Is this likely to be going to be f2f testing or will  participants 
>> simply publish remote endpoints?
> 
> 
> We'll have to figure that out; it's pretty much up to us. In past W3C 
> WGs I've participated in, people have self-submitted their test 
> results; i.e., they're not verified or tested for interop, just for 
> feature coverage and correctness. We can certainly do interop testing 
> if we like, of course; as Philippe has mentioned, we could hold an 
> internop workshop to do more.
> 
>> 2. If the procedure is likely to be that the participants publish 
>> remote endpoints (without need for meeting), what sort of timescale 
>> is usually expected for this testing phase?  For example, would 
>> participants have a few weeks to run tests and resolve problems or 
>> would the interop be more intense - such as a week focussed on 
>> problem resolution and testing?
> 
> 
> Don't know yet. In past interop testing efforts I've been involved  in, 
> we had a number of small, intense and focused interop mini- sessions 
> leading up to a bigger event.
> 
> The straw-man that I have in mind is a (semi-sequential) list of  things 

> that need to happen; if we had discussion around it, it might  help 
> answer these questions.
> 
> 1. Agree on and document testable features, and their optionality
> 2. Agree on and document test targets (e.g., service instance,  service 
> consumer)
> 3. Design a test scenario for each feature/target combination as 
> applicable, with success criteria
> 4. Hold a number of virtual (i.e., over the net) interop sessions 
> around specific features in isolation
> 5. Give feedback to implementors / specification from interop testing
> 6. Hold one F2F interop event testing features together (probably NOT  a 

> normal meeting of the WG)
> 7. Documentation of interop results to exit CR
> 
> Comments?
> 
> Cheers,
> 
> -- 
> Mark Nottingham   Principal Technologist
> Office of the CTO   BEA Systems
> 
>