W3C home > Mailing lists > Public > public-qt-comments@w3.org > September 2006

[Bug 3759] [F+O] round-half-to-even and float precision

From: <bugzilla@wiggum.w3.org>
Date: Thu, 21 Sep 2006 08:05:16 +0000
To: public-qt-comments@w3.org
Message-Id: <E1GQJYa-0000f6-Sk@wiggum.w3.org>


           Summary: [F+O] round-half-to-even and float precision
           Product: XPath / XQuery / XSLT
           Version: Candidate Recommendation
          Platform: PC
        OS/Version: Windows XP
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Functions and Operators
        AssignedTo: ashok.malhotra@oracle.com
        ReportedBy: mike@saxonica.com
         QAContact: public-qt-comments@w3.org

This issue arises from an XSLT test case bug:


but it is not specific to XSLT. The question is, what exactly does it mean in
F+O 6.4.5 when it says "If two such values are equally near (e.g. if the
fractional part in $arg is exactly .500...), returns the one whose least
significant digit is even. "

The case in question is 

round-half-to-even(xs:float(150.0150e0), 2)

At first sight 150.015 is equally close to 150.01 and 150.02, so it should be
rounded up. However, xs:float(150.015e0) actually returns a number whose
precise value is 150.0149993896484375, which would suggest rounding down. (This
is what I think should be done, and what the published test results do). 

However, Joanne has argued:

xs:float(150.01) = 150.00999450683594
xs:float(150.015) = 150.01499938964844
xs:float(150.02) = 150.02000427246094

and therefore 150.015 is equally close to both. But I can't see any
justification for this line of reasoning in the spec. The spec says we should
compare with "a multiple of ten to the power of minus $precision.", not with an
xs:float approximation to this.
Received on Thursday, 21 September 2006 08:05:26 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 16:57:15 UTC