[Bug 3529] [F&O] digits of precision returned by the numeric operators

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3529

           Summary: [F&O] digits of precision returned by the numeric
                    operators
           Product: XPath / XQuery / XSLT
           Version: Candidate Recommendation
          Platform: PC
        OS/Version: Windows XP
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Functions and Operators
        AssignedTo: ashok.malhotra@oracle.com
        ReportedBy: andrew.eisenberg@us.ibm.com
         QAContact: public-qt-comments@w3.org


Section 6.2, Operators on Numeric Values, states in part:

"For xs:decimal values the number of digits of precision returned by the
numeric operators is ·implementation-defined·. If the number of digits in the
result exceeds the number of digits that the implementation supports, the
result is truncated or rounded in an ·implementation-defined· manner."

Consider the query "1 div 3". Certainly a result of 0.333333333333333333 (18
digits) would be permitted. Would a value of 0.333333333333333 (15 digits) be
permitted as well?

Michael Kay has argued:

"The first sentence must be read in the context of the second, which creates a
relationship between the number of digits in the result of decimal division and
the number of digits supported by the xs:decimal data type. The latter must be
at least 18, and I believe that the result of division should therefore not
truncate or round unless it requires more than 18 digits."

This question is being raised based on several test cases, as reported on the
XML Query Test Suite in [1] and [2].


[1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=3443
[2] http://www.w3.org/Bugs/Public/show_bug.cgi?id=3528

Received on Monday, 24 July 2006 21:04:51 UTC