Re: results of MS validator run against cosmos tests downloaded on 1/15/09

The test plan only gets updated at iteration build boundaries (and there 
might be even more process hung on it than that, I'm not sure).  I can run 
the generator against a workspace containing all the patches from my 
previous email and mail the output, if it helps.  The next planned 
iteration is in early Feb (2/4 code cutoff IIRC).

+1: When a test is incorrect, the result returned by an implementation 
cannot really be trusted.
It is also true that a test case need only be uniquely identifiable + 
resolvable in order to be unambiguously discussed and run through code. It 
need not reside in CVS at that particular point in time, although having 
them all in one place certainly reduces the complexity.  You know the 
saying: you can have it cheap, fast, or simple ... choose any two.

My previous email contained the URLs (2 bugs + 1 email) where the 
revised/new test cases could be found precisely because there is (always) 
a delay + work in getting them integrated.  In cases where SML commented 
on existing tests and the COSMOS folks agreed to make the changes, the 
revisions on the integration queue are found in the bugs (that is what the 
committers pull from).  In cases where SML asked for new test cases, 
COSMOS is not going to integrate them until the known implementers agree 
that they are valid (short-circuiting at least one run of the "oops" loop 
& its delays).  Sandy and I have reviewed them for IBM; if Microsoft finds 
no problems, they can be queued for integration into CVS.  If you do find 
problems, please just state the problems (do not update them to fix the 
problems, or say what those updates should be, so the IP is clean).

Best Regards, John

Street address: 2455 South Road, P328 Poughkeepsie, NY USA 12601
Voice: 1+845-435-9470      Fax: 1+845-432-9787



From:
Kumar Pandit <kumarp@windows.microsoft.com>
To:
"public-sml@w3.org" <public-sml@w3.org>
Cc:
Len Charest <Len.Charest@microsoft.com>, Kumar Pandit 
<kumarp@windows.microsoft.com>
Date:
01/20/2009 03:23 AM
Subject:
results of MS validator run against cosmos tests downloaded on  1/15/09
Sent by:
public-sml-request@w3.org



I downloaded tests with remarks (such as ‘incorrect test’, ‘not yet 
integrated’, etc.) and ran the MS SML validator against the entire set of 
tests. Although I downloaded the tests on 1/15, I verified that the 
relevant tests have not changed on 1/19. 
 
Some observations:
 
n  Incorrect tests: I re-added ‘incorrect test’ remarks to the tests that 
have not been corrected as on 1/19 in the cosmos cvs repository. When a 
test is incorrect, the result returned by an implementation cannot really 
be trusted. I have changed the ‘MS’ column value from NA to I in such 
cases. I did not change the ‘cosmos’ column in this way. 
n  ‘ms: needs investigation’ tests: Investigated and updated results for 
the following tests. 
o    base64\Base64InvalidSML.xml
o    references\InValidSchemeResult.xml
o    targetRequired\InValidMissingReference.xml
n  ‘ms: needs investigation’ tests: Some of the tests in this category got 
converted to ‘incorrect test’ category:
o    InValidConstraintRefKey.xml
o    InValidDuplicateKeyConstraintName.xml
o    rules\invalid.xml
o    rules\InValidRuleType.xml
o    rules\InValidRuleTypeExtension.xml
o    rules\InValidRuleTypeRestriction.xml
n  I also added all of the ‘ms: need to get test’ set of tests (total 11).
n  new tests integrated
o    acyclic\ValidateDefinition.xml
o    acyclic\ValidDerivation.xml
 
None of the newly added tests (rows 144 ~ 152) are present in cosmos (
http://wiki.eclipse.org/COSMOS_SML_Test_Plan#Test_Suite) they were not run 
against the ms validator.
 
I have checked in the updated xlsx file into cvs. Nonetheless, here is a 
copy for quick reference.
 
 [attachment "sml-test-v4.zip" deleted by John Arwe/Poughkeepsie/IBM]