RE: results of MS validator run against cosmos tests downloaded on 1/15/09

The urls sent by John have eclipse patch files instead of full updated files. I do not have/use the necessary tools to generate the final files from the patches therefore I could not use them earlier. John sent me zip files with the updated test cases on another email. I unpacked the test cases and ran the MS SML validator against them. I have checked in the result of that run.

The test 'ValidBareNameDTDDeterminedIDBareName.xml' was marked as 'R' (required feature), I changed to 'O' (optional feature).

See inline comments for the rest.

From: public-sml-request@w3.org [mailto:public-sml-request@w3.org] On Behalf Of John Arwe
Sent: Tuesday, January 20, 2009 10:37 AM
To: public-sml@w3.org
Subject: Re: results of MS validator run against cosmos tests downloaded on 1/15/09


Looking at the CVS 1.4 version (latest at the moment, corresponding to Kumar's email), we seem to be drifting off of earlier discussions about test results (as a binary condition of "validity") and into "correctness" (invalid for "the right reasons") land.  Remember that we decided months ago that the latter part was out of scope... people convinced me!

16, 17: CVS is not the latest version, latest is in a cosmos bug awaiting integration by a committer

[kumarp] status updated. Updated tests are incorrect.

29: It's hard for me to call the test incorrect.  It's fine if the validator recognizes the xpointer() scheme.  SML would either need to add that as a new feature, or simply remove this test from the "relevant for interop" set as it did with the test cases with multiple sml:uri children in a single SML reference.

[kumarp] This test uses xpointer() fragment under sml:uri. The SML spec allows only smlxpath1() or barename fragement under sml:uri. The above statement holds if the xpointer() fragment is present under some element other than sml:uri.

35, 39: Based on earlier discussions this seems like an informational comment: since our test plan states that expected results are phrased in terms of SML(IF)-validity, i.e. they are binary, the expected and actual results are still the same.  If the Microsoft implementation reports the model as invalid, that is a fact.  As we discussed back in November, each implementation probably wants to (in the case of "not valid" results) ensure it labels the model as invalid for "the right" reason(s), but that is irrelevant for CfI purposes as the working group has defined things.  That said, I suspect COSMOS will happily change these "unintended" test cases and will communicate the SML wg remarks to them as in the past.

[kumarp] This is a bit tricky. The MS validator does not declare the model as invalid. It is declared as un-processable because the id constraint is incorrectly encoded. That is, ':' is used when an NCNAME is expected. I am ok if the WG decides to accept this as the expected result. It may be easier to just remove 'tns:' from the NCNAME to make the test correct.

44: CVS is not the latest version, ...

[kumarp] status updated. Updated test is still incorrect. It has multiple schema errors.

55, 56: "1. incorrect test: documentation uses tags such as <p>, <code>, etc. that are not defined. , 2. check if rules are on type or in doc"  wrt #1, why does this effect the outcome of the validation test?  The content model for xs:annotation/xs:documentation is xs:any 0..* according to Structures 3.13.  wrt #2, they are rule documents in both cases, but I'm not sure why this remark is intended to assert/convey.

[kumarp] I am not sure of the right behavior here. The documentation uses some elements from the xhtml namespace whose schema is not included in the SML-IF. This causes schema validation error for those tags. Can you clarify if the documentation tags are exempt from schema validation according to the spec?

74: we can remove the remark if we like I think, since the primary feature is now == validation.

[kumarp] agreed. remark removed.

88: Since COSMOS contains code to actually test this case, it would be a regression from their point of view to change it as suggested.  SML either needs to declare the SML-validity results to be whatever results from 0 reference scheme instances as it is currently coded (and let COSMOS have different results), or declare this test case as broken/uninteresting for interop since it uses the double-sml:uri syntax.  When we ran into its multi-sml:uri brethren earlier, SML simply stopped pointing at them for interop, so doing so in this case would be consistent with past decisions.

[kumarp] I am ok with removing this test from the interop suite.

100: Again, the remarks (and resulting MS=I value) appear to be resulting from drift between SML validity results (in scope) and the -reason- for the model being viewed as invalid (out of scope for CfI).  That aside, where in our spec does it say this case is not allowed?  I saw nothing about it in SML chapter 6 when I tried to find a reference to point the COSMOS folks at.  Ditto Schematron.

[kumarp] SML spec section 6.1: "The context expression is defined as an XSLT Pattern. This means that the smlfn:deref function may not be used in the location path of a context expression." If the WG agrees that incorrectly encoded SML model == invalid model then I am ok with this test being used for the purpose, otherwise we need to decide whether to fix this test or remove it from the interop suite.

101-105: As I pointed out before, "NA" is not really true.  For an otherwise valid model (i.e. one for which only the locid value itself, when used, causes a problem), an implementation that does not use that value should report valid.  Invalid... invalid.  This should be entirely predictable.

[kumarp] Agreed that MS validator can run these test cases without supporting locid. I have added the tests to the set of tests run against MS validator. I found that all of them use deref() in the context expression, therefore my earlier comment applies to the 5 tests. I have updated xlsx with this info.

111-114: very similar to 100

144-152: test case source was/is attached to the email http://lists.w3.org/Archives/Public/public-sml/2009Jan/0013.html

[kumarp] added all of the tests. They get the expected output against MS SML validator. Updated xlsx.

Best Regards, John

Street address: 2455 South Road, P328 Poughkeepsie, NY USA 12601
Voice: 1+845-435-9470      Fax: 1+845-432-9787