Re: Action A-659-05 Completed

> 
> The lists below are the test cases which are either failed by all implementations or only passed by one implementation. 

A little more analysis on these:
> 
> 
> Failed by all implementations:
> 
> 
> ST-Data001 (static typing feature)

To be honest, there is only one implementation passing any of the static typing tests, namely XMLPrime, but it comes up in XPath, XQuery, and XQueryX guises. I'm not sure why this particular test differs from the other static typing tests.

> collation-key-009u (probably failing due to the new dependency added)


>  compare-031 (probably failing due to the new dependency added)
> compare-034 (probably failing due to the new dependency added)
> compare-035 (probably failing due to the new dependency added)
> compare-039 (probably failing due to the new dependency added)

These are UCA collation tests. I think Saxon should be able to pass these with a tweak to the way the test driver handles dependencies.  Perhaps the same is true of XMLPrime, but we may never know. These are quite tough tests of UCA collations, added fairly recently so implementors haven't had much time to respond to them.
> 
> 
> Passed by one implementation only:
> 
> substitution-020 (XML schema 1.1 dependency)
> substitution-021 (XML schema 1.1 dependency)
> substitution-022 (XML schema 1.1 dependency)
> substitution-023 (XML schema 1.1 dependency)
> substitution-024 (XML schema 1.1 dependency)
> substitution-025 (XML schema 1.1 dependency)
> Serialization-adaptive-37 (XML schema 1.1 dependency)

These tests require support for XSD 1.1, which only Saxon (among the implementations that have reported results) currently supports.
> 
> 
> Currently only two implementations support fn:transform. The following fn:transform tests were only passed by one implementation. It looks like we are still waiting on features to be implemented.

Actually we have three PRODUCTS reporting some test successes for fn:transform (Exselt, XMLPrime, Saxon). For some of the cases where tests were passed by two implementations, the implementations in question are Saxon XQuery and Saxon XPath, therefore not independent. Many of the tests in the list below have no result for XMLPrime, which may be because the tests were created rather recently in response to late spec changes.
> 
> fn-transform-2
> fn-transform-19
> fn-transform-37
> fn-transform-50
> fn-transform-51
> fn-transform-52
> fn-transform-53
> fn-transform-54
> fn-transform-55
> fn-transform-56
> fn-transform-57
> fn-transform-58
> fn-transform-59
> fn-transform-60
> fn-transform-61
> fn-transform-62
> fn-transform-63
> fn-transform-64
> fn-transform-66
> 

So overall the situation is pretty healthy given that we are talking about a few dozen problematic tests out of 30,000. But in some respects we are over-egging the results; the figures don't distinguish whether the implementations that are passing the tests are truly independent of each other.

We can summarize the analysis by saying that there appear to be four areas where reported test results show that implementation is imcomplete:

* Support for static typing. This is a feature that we retain in the specification for backwards compatibility, but which has fallen out of favour with many implementors. It was always an optional feature. I don't think there is any practical alternative to our approach here, of retaining support for the feature in 3.1 given that it is supported in some products.

* Support for XSD 1.1. Most vendors of XSLT and XQuery processors rely on a third party implementation of XSD (Saxonica and Altova are notable exceptions) and many XSD 1.0 processors have not been upgraded to XSD 1,1. Again there appears to be no practical alternative to our approach of making support for XSD 1.1 optional.

* Support for UCA Collations. We chose to make support for UCA collations mandatory in the interests of interoperability and I18N support, but implementation is technically challenging, and typically depends on third party libraries. We could make the feature optional, but we don't think this would be in the interests of the user community; rather we would prefer to accept that implementation of these features may lag a little behind.

* Support for fn:transform() (dynamic invocation of XSLT from XQuery). There is strong evidence that users require this feature, but it is difficult to ensure 100% interoperability of implementations because in many cases it will depend on integrating a third-party product. Tests where the transformation requires XSLT 3.0 are particularly challenging because the standard is so new. We think we have done the right thing by including this feature although it will probably remain true that different vendors will offer different subsets of the capability.

Michael Kay
Saxonica