- From: Jonathan Marsh <jonathan@wso2.com>
- Date: Tue, 5 Dec 2006 10:30:12 -0800
- To: "'Arthur Ryman'" <ryman@ca.ibm.com>, "'Lawrence Mandel'" <lmandel@ca.ibm.com>
- Cc: "'www-ws-desc'" <www-ws-desc@w3.org>, <www-ws-desc-request@w3.org>
- Message-ID: <007c01c7189b$69576080$3401a8c0@DELLICIOUS>
If an assertion is always masked by a precondition, then it doesn't make much sense to test as an independent assertion. Conversely, if a test document violates a precondition as well as the main assertion, it seems to be a sub-standard unit test for that assertion. So ideally, we should be able to write a set of test documents that violate one and only one assertion. In practice that might be tough though. I'd concentrate on filling out the untested assertions and getting rid of yellow before quibbling over the how green a successful test should be ;-). Jonathan Marsh - <http://www.wso2.com> http://www.wso2.com - <http://auburnmarshes.spaces.live.com> http://auburnmarshes.spaces.live.com _____ From: Arthur Ryman [mailto:ryman@ca.ibm.com] Sent: Tuesday, December 05, 2006 6:18 AM To: Lawrence Mandel Cc: Jonathan Marsh; 'www-ws-desc'; www-ws-desc-request@w3.org Subject: RE: Validation results Lawrence, It would be very difficult to list the dependency information of each assertion, although I think that would be useful. What we need here is essentially the preconditions for each assertion. A validator should not bother to test an assertion if any of its preconditions fail. So this is really not something to add to the test suite, but rather to the spec itself. Each assertion in the spec should have a list of precondition assertions. I suppose we could do this incrementally to start, i.e. when we find an assertion that depends on others being satisfied, then list them as preconditions. This information would be provided to validators in order to eliminate spurious violation reports. Arthur Ryman, IBM Software Group, Rational Division blog: http://ryman.eclipsedevelopersjournal.com/ phone: +1-905-413-3077, TL 969-3077 assistant: +1-905-413-2411, TL 969-2411 fax: +1-905-413-4920, TL 969-4920 mobile: +1-416-939-5063, text: 4169395063@fido.ca Lawrence Mandel/Toronto/IBM@IBMCA Sent by: www-ws-desc-request@w3.org 12/05/2006 08:30 AM To Arthur Ryman/Toronto/IBM@IBMCA cc "Jonathan Marsh" <jonathan@wso2.com>, "'www-ws-desc'" <www-ws-desc@w3.org>, www-ws-desc-request@w3.org Subject RE: Validation results >It may report others since in some cases an assertion doesn't make sense unless other assertions are satisfied. In this case, where multiple assertions are violated, I think the test case should list all of the violated assertions. If this is the case the test should not show green unless the implementation reports all the assertion violations and only the assertion violations listed in the test case. Lawrence Mandel Software Developer IBM Rational Software Phone: 905 - 413 - 3814 Fax: 905 - 413 - 4920 lmandel@ca.ibm.com Arthur Ryman/Toronto/IBM 12/04/2006 06:13 PM To "Jonathan Marsh" <jonathan@wso2.com> cc Lawrence Mandel/Toronto/IBM@IBMCA, "'www-ws-desc'" <www-ws-desc@w3.org>, www-ws-desc-request@w3.org Subject RE: Validation resultsLink <Notes://d25ml04/85256CA30060A6BE/38D46BF5E8F08834852564B500129B2C/9F8BFC300 0B0DB198525723A007B0430> Jonathan, I agree that if the processor reports the expected assertion that it gets a green. It may report others since in some cases an assertion doesn't make sense unless other assertions are satisfied. Ideally, a processor should limit the number of assertion violations reported to eliminate ones that are consequences of errors. Arthur Ryman, IBM Software Group, Rational Division blog: http://ryman.eclipsedevelopersjournal.com/ phone: +1-905-413-3077, TL 969-3077 assistant: +1-905-413-2411, TL 969-2411 fax: +1-905-413-4920, TL 969-4920 mobile: +1-416-939-5063, text: 4169395063@fido.ca "Jonathan Marsh" <jonathan@wso2.com> Sent by: www-ws-desc-request@w3.org 12/04/2006 05:21 PM To Lawrence Mandel/Toronto/IBM@IBMCA cc "'www-ws-desc'" <www-ws-desc@w3.org> Subject RE: Validation results Yes, so far it's by design. Validation is considered successful if, among the identified assertions that have been violated is the one associated with the test case. It's not considered unsuccessful if it identifies additional assertions that have been violated - or multiple instances of violating a single assertion. However, I put that in place without much thought. It would not be hard to flag those results that weren't one-to-one matches with the identified assertion. Whatever makes most sense to Woden at this point. Arthur, what do you think? Jonathan Marsh - <http://www.wso2.com/> http://www.wso2.com - <http://auburnmarshes.spaces.live.com/> http://auburnmarshes.spaces.live.com _____ From: Lawrence Mandel [mailto:lmandel@ca.ibm.com] Sent: Monday, December 04, 2006 12:11 PM To: Jonathan Marsh Cc: 'www-ws-desc' Subject: RE: Validation results Jonathan, With your most recent change I see all green on tests such as Binding-3B, even though the test cases indicates that only assertion Binding-0057 should be violated and Woden identifies that assertions Binding-0057 and Binding-0055 have been violated. Is this by design? I would think assertion Binding-0055 should be flagged with a yellow or red marker. Lawrence Mandel Software Developer IBM Rational Software Phone: 905 - 413 - 3814 Fax: 905 - 413 - 4920 lmandel@ca.ibm.com "Jonathan Marsh" <jonathan@wso2.com> Sent by: www-ws-desc-request@w3.org 12/04/2006 01:29 PM To "'Youenn Fablet'" <youenn.fablet@crf.canon.fr> cc "'www-ws-desc'" <www-ws-desc@w3.org> Subject RE: Validation results Good catch, I've fixed it, with more green appearing as a result. Checked in your results too: http://dev.w3.org/cvsweb/~checkout~/2002/ws/desc/test-suite/results/Validati on.html Jonathan Marsh - http://www.wso2.com - http://auburnmarshes.spaces.live.com > -----Original Message----- > From: Youenn Fablet [mailto:youenn.fablet@crf.canon.fr] > Sent: Monday, December 04, 2006 8:39 AM > To: Jonathan Marsh; www-ws-desc > Subject: Validation results > > I have regenerated the validation results. > I have noted that the report is not quite accurate with the good > documents. > They are either flagged red or yellow, while they should be either red > or green. > Maybe a small change in the presentation stylesheet would greatly help > improving the overall results ;-) > Jonathan, can you update the CVS canon validation results file with this > one? > Thanks, > Youenn >
Received on Tuesday, 5 December 2006 18:41:52 UTC