[Bug 3759] [F+O] round-half-to-even and float precision

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3759

           Summary: [F+O] round-half-to-even and float precision
           Product: XPath / XQuery / XSLT
           Version: Candidate Recommendation
          Platform: PC
        OS/Version: Windows XP
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Functions and Operators
        AssignedTo: ashok.malhotra@oracle.com
        ReportedBy: mike@saxonica.com
         QAContact: public-qt-comments@w3.org


This issue arises from an XSLT test case bug:

http://www.w3.org/Member/bugzilla/show_bug.cgi?id=624
(member-only)

but it is not specific to XSLT. The question is, what exactly does it mean in
F+O 6.4.5 when it says "If two such values are equally near (e.g. if the
fractional part in $arg is exactly .500...), returns the one whose least
significant digit is even. "

The case in question is 

round-half-to-even(xs:float(150.0150e0), 2)

At first sight 150.015 is equally close to 150.01 and 150.02, so it should be
rounded up. However, xs:float(150.015e0) actually returns a number whose
precise value is 150.0149993896484375, which would suggest rounding down. (This
is what I think should be done, and what the published test results do). 

However, Joanne has argued:

xs:float(150.01) = 150.00999450683594
xs:float(150.015) = 150.01499938964844
xs:float(150.02) = 150.02000427246094

and therefore 150.015 is equally close to both. But I can't see any
justification for this line of reasoning in the spec. The spec says we should
compare with "a multiple of ten to the power of minus $precision.", not with an
xs:float approximation to this.

Received on Thursday, 21 September 2006 08:05:26 UTC