RE: Validation results

Jonathan,

An assertion may not make sense unless other assertions are satisfied. 
That doesn't mean it's always masked. Just like when you compile a 
program, a single bad line of code might generate tons of other error 
messages. Those other messages don't help.

I'll analyse the specific cases of where Woden is reporting multiple 
violations to see if this is in fact the case.

Arthur Ryman,
IBM Software Group, Rational Division

blog: http://ryman.eclipsedevelopersjournal.com/
phone: +1-905-413-3077, TL 969-3077
assistant: +1-905-413-2411, TL 969-2411
fax: +1-905-413-4920, TL 969-4920
mobile: +1-416-939-5063, text: 4169395063@fido.ca



"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org
12/05/2006 01:30 PM

To
Arthur Ryman/Toronto/IBM@IBMCA, Lawrence Mandel/Toronto/IBM@IBMCA
cc
"'www-ws-desc'" <www-ws-desc@w3.org>, <www-ws-desc-request@w3.org>
Subject
RE: Validation results






If an assertion is always masked by a precondition, then it doesn?t make 
much sense to test as an independent assertion.
 
Conversely, if a test document violates a precondition as well as the main 
assertion, it seems to be a sub-standard unit test for that assertion.
 
So ideally, we should be able to write a set of test documents that 
violate one and only one assertion.
 
In practice that might be tough though. I?d concentrate on filling out the 
untested assertions and getting rid of yellow before quibbling over the 
how green a successful test should be ;-).
 
Jonathan Marsh - http://www.wso2.com - 
http://auburnmarshes.spaces.live.com
 

From: Arthur Ryman [mailto:ryman@ca.ibm.com] 
Sent: Tuesday, December 05, 2006 6:18 AM
To: Lawrence Mandel
Cc: Jonathan Marsh; 'www-ws-desc'; www-ws-desc-request@w3.org
Subject: RE: Validation results
 

Lawrence, 

It would be very difficult to list the dependency information of each 
assertion, although I think that would be useful. 

What we need here is essentially the preconditions for each assertion. A 
validator should not bother to test an assertion if any of its 
preconditions fail. So this is really not something to add to the test 
suite, but rather to the spec itself. Each assertion in the spec should 
have a list of precondition assertions. 

I suppose we could do this incrementally to start, i.e. when we find an 
assertion that depends on others being satisfied, then list them as 
preconditions. This information would be provided to validators in order 
to eliminate spurious violation reports. 

Arthur Ryman,
IBM Software Group, Rational Division

blog: http://ryman.eclipsedevelopersjournal.com/
phone: +1-905-413-3077, TL 969-3077
assistant: +1-905-413-2411, TL 969-2411
fax: +1-905-413-4920, TL 969-4920
mobile: +1-416-939-5063, text: 4169395063@fido.ca 


Lawrence Mandel/Toronto/IBM@IBMCA 
Sent by: www-ws-desc-request@w3.org 
12/05/2006 08:30 AM 


To
Arthur Ryman/Toronto/IBM@IBMCA 
cc
"Jonathan Marsh" <jonathan@wso2.com>, "'www-ws-desc'" 
<www-ws-desc@w3.org>, www-ws-desc-request@w3.org 
Subject
RE: Validation results
 


 
 





>It may report others since in some cases an assertion doesn't make sense 
unless other assertions are satisfied. 

In this case, where multiple assertions are violated, I think the test 
case should list all of the violated assertions. If this is the case the 
test should not show green unless the implementation reports all the 
assertion violations and only the assertion violations listed in the test 
case. 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 

Arthur Ryman/Toronto/IBM 
12/04/2006 06:13 PM 
 


To
"Jonathan Marsh" <jonathan@wso2.com> 
cc
Lawrence Mandel/Toronto/IBM@IBMCA, "'www-ws-desc'" <www-ws-desc@w3.org>, 
www-ws-desc-request@w3.org 
Subject
RE: Validation resultsLink
 


 
 




Jonathan, 

I agree that if the processor reports the expected assertion that it gets 
a green. It may report others since in some cases an assertion doesn't 
make sense unless other assertions are satisfied. 

Ideally, a processor should limit the number of assertion violations 
reported to eliminate ones that are consequences of errors. 

Arthur Ryman,
IBM Software Group, Rational Division

blog: http://ryman.eclipsedevelopersjournal.com/
phone: +1-905-413-3077, TL 969-3077
assistant: +1-905-413-2411, TL 969-2411
fax: +1-905-413-4920, TL 969-4920
mobile: +1-416-939-5063, text: 4169395063@fido.ca 

"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org 
12/04/2006 05:21 PM 
 


To
Lawrence Mandel/Toronto/IBM@IBMCA 
cc
"'www-ws-desc'" <www-ws-desc@w3.org> 
Subject
RE: Validation results
 


 
 





Yes, so far it?s by design.  Validation is considered successful if, among 
the identified assertions that have been violated is the one associated 
with the test case.  It?s not considered unsuccessful if it identifies 
additional assertions that have been violated ? or multiple instances of 
violating a single assertion. 
 
However, I put that in place without much thought.  It would not be hard 
to flag those results that weren?t one-to-one matches with the identified 
assertion.  Whatever makes most sense to Woden at this point? 
 
Arthur, what do you think? 
 
Jonathan Marsh - http://www.wso2.com - 
http://auburnmarshes.spaces.live.com 
 
 



From: Lawrence Mandel [mailto:lmandel@ca.ibm.com] 
Sent: Monday, December 04, 2006 12:11 PM
To: Jonathan Marsh
Cc: 'www-ws-desc'
Subject: RE: Validation results 
 

Jonathan, 

With your most recent change I see all green on tests such as Binding-3B, 
even though the test cases indicates that only assertion Binding-0057 
should be violated and Woden identifies that assertions Binding-0057 and 
Binding-0055 have been violated. Is this by design? I would think 
assertion Binding-0055 should be flagged with a yellow or red marker. 

Lawrence Mandel

Software Developer
IBM Rational Software
Phone: 905 - 413 - 3814   Fax: 905 - 413 - 4920
lmandel@ca.ibm.com 

"Jonathan Marsh" <jonathan@wso2.com> 
Sent by: www-ws-desc-request@w3.org 
12/04/2006 01:29 PM 
 


To
"'Youenn Fablet'" <youenn.fablet@crf.canon.fr> 
cc
"'www-ws-desc'" <www-ws-desc@w3.org> 
Subject
RE: Validation results


 
 


 
 







Good catch, I've fixed it, with more green appearing as a result.  Checked
in your results too:

http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Validati

on.html

Jonathan Marsh - http://www.wso2.com - 
http://auburnmarshes.spaces.live.com


> -----Original Message-----
> From: Youenn Fablet [mailto:youenn.fablet@crf.canon.fr]
> Sent: Monday, December 04, 2006 8:39 AM
> To: Jonathan Marsh; www-ws-desc
> Subject: Validation results
> 
> I have regenerated the validation results.
> I have noted that the report is not quite accurate with the good
> documents.
> They are either flagged red or yellow, while they should be either red
> or green.
> Maybe a small change in the presentation stylesheet would greatly help
> improving the overall results ;-)
> Jonathan, can you update the CVS canon validation results file with this
> one?
> Thanks,
>     Youenn
>