RE: Validation results

If an assertion is always masked by a precondition, then it doesn't make
much sense to test as an independent assertion.

 

Conversely, if a test document violates a precondition as well as the main
assertion, it seems to be a sub-standard unit test for that assertion.

 

So ideally, we should be able to write a set of test documents that violate
one and only one assertion.

 

In practice that might be tough though. I'd concentrate on filling out the
untested assertions and getting rid of yellow before quibbling over the how
green a successful test should be ;-).

 

Jonathan Marsh -  <http://www.wso2.com> http://www.wso2.com -
<http://auburnmarshes.spaces.live.com> http://auburnmarshes.spaces.live.com

 

  _____  

From: Arthur Ryman [mailto:ryman@ca.ibm.com] 
Sent: Tuesday, December 05, 2006 6:18 AM
To: Lawrence Mandel
Cc: Jonathan Marsh; 'www-ws-desc'; www-ws-desc-request@w3.org
Subject: RE: Validation results

 


Lawrence, 

It would be very difficult to list the dependency information of each
assertion, although I think that would be useful. 

What we need here is essentially the preconditions for each assertion. A
validator should not bother to test an assertion if any of its preconditions
fail. So this is really not something to add to the test suite, but rather
to the spec itself. Each assertion in the spec should have a list of
precondition assertions. 

I suppose we could do this incrementally to start, i.e. when we find an
assertion that depends on others being satisfied, then list them as
preconditions. This information would be provided to validators in order to
eliminate spurious violation reports. 

Arthur Ryman,
IBM Software Group, Rational Division

blog: http://ryman.eclipsedevelopersjournal.com/
phone: +1-905-413-3077, TL 969-3077
assistant: +1-905-413-2411, TL 969-2411
fax: +1-905-413-4920, TL 969-4920
mobile: +1-416-939-5063, text: 4169395063@fido.ca 




Lawrence Mandel/Toronto/IBM@IBMCA 
Sent by: www-ws-desc-request@w3.org 

12/05/2006 08:30 AM 


To

Arthur Ryman/Toronto/IBM@IBMCA 


cc

"Jonathan Marsh" <jonathan@wso2.com>, "'www-ws-desc'" <www-ws-desc@w3.org>,
www-ws-desc-request@w3.org 


Subject

RE: Validation results

 


 

 





>It may report others since in some cases an assertion doesn't make sense
unless other assertions are satisfied. 

In this case, where multiple assertions are violated, I think the test case
should list all of the violated assertions. If this is the case the test
should not show green unless the implementation reports all the assertion
violations and only the assertion violations listed in the test case. 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 


Arthur Ryman/Toronto/IBM 

12/04/2006 06:13 PM 

 


To

"Jonathan Marsh" <jonathan@wso2.com> 


cc

Lawrence Mandel/Toronto/IBM@IBMCA, "'www-ws-desc'" <www-ws-desc@w3.org>,
www-ws-desc-request@w3.org 


Subject

RE: Validation resultsLink
<Notes://d25ml04/85256CA30060A6BE/38D46BF5E8F08834852564B500129B2C/9F8BFC300
0B0DB198525723A007B0430> 

 


 

 




Jonathan, 

I agree that if the processor reports the expected assertion that it gets a
green. It may report others since in some cases an assertion doesn't make
sense unless other assertions are satisfied. 

Ideally, a processor should limit the number of assertion violations
reported to eliminate ones that are consequences of errors. 

Arthur Ryman,
IBM Software Group, Rational Division

blog: http://ryman.eclipsedevelopersjournal.com/
phone: +1-905-413-3077, TL 969-3077
assistant: +1-905-413-2411, TL 969-2411
fax: +1-905-413-4920, TL 969-4920
mobile: +1-416-939-5063, text: 4169395063@fido.ca 


"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org 

12/04/2006 05:21 PM 

 


To

Lawrence Mandel/Toronto/IBM@IBMCA 


cc

"'www-ws-desc'" <www-ws-desc@w3.org> 


Subject

RE: Validation results

 


 

 





Yes, so far it's by design.  Validation is considered successful if, among
the identified assertions that have been violated is the one associated with
the test case.  It's not considered unsuccessful if it identifies additional
assertions that have been violated - or multiple instances of violating a
single assertion. 
 
However, I put that in place without much thought.  It would not be hard to
flag those results that weren't one-to-one matches with the identified
assertion.  Whatever makes most sense to Woden at this point. 
 
Arthur, what do you think? 
 
Jonathan Marsh -  <http://www.wso2.com/> http://www.wso2.com -
<http://auburnmarshes.spaces.live.com/> http://auburnmarshes.spaces.live.com

  

 

  _____  



From: Lawrence Mandel [mailto:lmandel@ca.ibm.com] 
Sent: Monday, December 04, 2006 12:11 PM
To: Jonathan Marsh
Cc: 'www-ws-desc'
Subject: RE: Validation results 
 

Jonathan, 

With your most recent change I see all green on tests such as Binding-3B,
even though the test cases indicates that only assertion Binding-0057 should
be violated and Woden identifies that assertions Binding-0057 and
Binding-0055 have been violated. Is this by design? I would think assertion
Binding-0055 should be flagged with a yellow or red marker. 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 


"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org 

12/04/2006 01:29 PM 

 


To

"'Youenn Fablet'" <youenn.fablet@crf.canon.fr> 


cc

"'www-ws-desc'" <www-ws-desc@w3.org> 


Subject

RE: Validation results



  

 


  

 







Good catch, I've fixed it, with more green appearing as a result.  Checked
in your results too:

http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Validati
on.html

Jonathan Marsh - http://www.wso2.com - http://auburnmarshes.spaces.live.com


> -----Original Message-----
> From: Youenn Fablet [mailto:youenn.fablet@crf.canon.fr]
> Sent: Monday, December 04, 2006 8:39 AM
> To: Jonathan Marsh; www-ws-desc
> Subject: Validation results
> 
> I have regenerated the validation results.
> I have noted that the report is not quite accurate with the good
> documents.
> They are either flagged red or yellow, while they should be either red
> or green.
> Maybe a small change in the presentation stylesheet would greatly help
> improving the overall results ;-)
> Jonathan, can you update the CVS canon validation results file with this
> one?
> Thanks,
>     Youenn
>