RE: Requested enhancement to assertion coverage report

OK, makes sense.  Checked it in.

 

http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/Assertion-covera
ge.html

 

 

Jonathan Marsh -  <http://www.wso2.com> http://www.wso2.com -
<http://auburnmarshes.spaces.live.com> http://auburnmarshes.spaces.live.com

 

  _____  

From: Lawrence Mandel [mailto:lmandel@ca.ibm.com] 
Sent: Monday, November 27, 2006 9:03 PM
To: Jonathan Marsh; www-ws-desc@w3.org
Subject: RE: Requested enhancement to assertion coverage report

 


I thought it would be useful to see if all of our bad test cases are simple
and fail a single assertion or if there are many that fail multiple
assertions. (Real world test cases that I've seen for WSDL 1.1 often fail
multiple assertions.) I think it would also be good to see if there are
certain assertions that always fail together to determine if any dependent
assertions have been missed. 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 




"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org 

11/27/2006 11:22 PM 


To

Lawrence Mandel/Toronto/IBM@IBMCA, <www-ws-desc@w3.org> 


cc

 


Subject

RE: Requested enhancement to assertion coverage report

 


 

 




How does that help?  I can see a roll-up of how complex good testcases are,
but what's the difference in practice between a document that violates one
assertion and one that violates two? 
  
Jonathan Marsh -  <http://www.wso2.com/> http://www.wso2.com -
<http://auburnmarshes.spaces.live.com/> http://auburnmarshes.spaces.live.com

  

 

  _____  


From: www-ws-desc-request@w3.org [mailto:www-ws-desc-request@w3.org] On
Behalf Of Lawrence Mandel
Sent: Thursday, November 16, 2006 11:47 PM
To: www-ws-desc@w3.org
Subject: Re: Requested enhancement to assertion coverage report 
  

One further request. (Knew I forgot one when I clicked send.) 

3. I'd like to see the complexity of the test documents in the test suite.
I've focused on creating very simple tests that for the most part violate
one assertion. I think it would be useful to understand the complexity of
the documents in the suite by viewing how many violate 1 assertion, 2
assertions and 3 or more assertions. I'm thinking of a section such as 

Complexity of Test Documents: 
# docs that violate 1 assertion: 80 
# docs that violate 2 assertions: 10 
# docs that violate 3 or more assertions: 2 

Thanks again, 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 


Lawrence Mandel/Toronto/IBM@IBMCA 
Sent by: www-ws-desc-request@w3.org 

11/17/2006 02:36 AM 

 


To

www-ws-desc@w3.org 


cc

  


Subject

Requested enhancement to assertion coverage report


  

 


  

 






Jonathan, 

First of all, nice work on the reports. 

I have a couple requests, assuming you can find the time. :) 

1. Given that you found a couple tests that referenced invalid assertion
IDs, I'd like to request that a section be added to the assertion report
that shows the tests that are referencing invalid assertion IDs. I think
this will be useful as assertions are removed from the spec and when the
assertions are renumbered. 

2. It would also be nice to know (this really is only a nice to have item on
the importance scale) the coverage of part 1 and part 2 of the spec. I'd
like to request that the numbers be included in the totals summary. For
example, 

98/228 assertions (60/157 component, 20/50 document, 18/18 exchange, 0/3
message). 80/106 for part 1, 18/122 for part 2. 

Thanks, 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com