RE: Requested enhancement to assertion coverage report

I thought it would be useful to see if all of our bad test cases are 
simple and fail a single assertion or if there are many that fail multiple 
assertions. (Real world test cases that I've seen for WSDL 1.1 often fail 
multiple assertions.) I think it would also be good to see if there are 
certain assertions that always fail together to determine if any dependent 
assertions have been missed.

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com



"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org
11/27/2006 11:22 PM

To
Lawrence Mandel/Toronto/IBM@IBMCA, <www-ws-desc@w3.org>
cc

Subject
RE: Requested enhancement to assertion coverage report






How does that help?  I can see a roll-up of how complex good testcases 
are, but what?s the difference in practice between a document that 
violates one assertion and one that violates two?
 
Jonathan Marsh - http://www.wso2.com - 
http://auburnmarshes.spaces.live.com
 

From: www-ws-desc-request@w3.org [mailto:www-ws-desc-request@w3.org] On 
Behalf Of Lawrence Mandel
Sent: Thursday, November 16, 2006 11:47 PM
To: www-ws-desc@w3.org
Subject: Re: Requested enhancement to assertion coverage report
 

One further request. (Knew I forgot one when I clicked send.) 

3. I'd like to see the complexity of the test documents in the test suite. 
I've focused on creating very simple tests that for the most part violate 
one assertion. I think it would be useful to understand the complexity of 
the documents in the suite by viewing how many violate 1 assertion, 2 
assertions and 3 or more assertions. I'm thinking of a section such as 

Complexity of Test Documents: 
# docs that violate 1 assertion: 80 
# docs that violate 2 assertions: 10 
# docs that violate 3 or more assertions: 2 

Thanks again, 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 


Lawrence Mandel/Toronto/IBM@IBMCA 
Sent by: www-ws-desc-request@w3.org 
11/17/2006 02:36 AM 


To
www-ws-desc@w3.org 
cc
 
Subject
Requested enhancement to assertion coverage report
 


 
 





Jonathan, 

First of all, nice work on the reports. 

I have a couple requests, assuming you can find the time. :) 

1. Given that you found a couple tests that referenced invalid assertion 
IDs, I'd like to request that a section be added to the assertion report 
that shows the tests that are referencing invalid assertion IDs. I think 
this will be useful as assertions are removed from the spec and when the 
assertions are renumbered. 

2. It would also be nice to know (this really is only a nice to have item 
on the importance scale) the coverage of part 1 and part 2 of the spec. 
I'd like to request that the numbers be included in the totals summary. 
For example, 

98/228 assertions (60/157 component, 20/50 document, 18/18 exchange, 0/3 
message). 80/106 for part 1, 18/122 for part 2. 

Thanks, 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com