[Bug 3759] [F+O] round-half-to-even and float precision

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3759


mike@saxonica.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED




------- Comment #3 from mike@saxonica.com  2006-10-24 15:50 -------
The working group accepted the following proposal, which was made by email at
http://lists.w3.org/Archives/Member/w3c-xsl-query/2006Oct/0005.html
(member-only):

OPTION 2: define the function to work by converting the argument to decimal,
and then converting the result back to the original type. Below is suggested
text for that option.


<proposal>
fn:round-half-to-even($arg as numeric?) as numeric?
fn:round-half-to-even($arg as numeric?, $precision as xs:integer) as
numeric?

Summary: The value returned is the nearest (that is, numerically closest)
numeric to $arg that is a multiple of ten to the power of minus $precision.
If two such values are equally near (e.g. if the fractional part in $arg is
exactly .500...), the function returns the one whose least significant digit
is even. 

If type of $arg is one of the four numeric types xs:float, xs:double,
xs:decimal or xs:integer the type of the result is the same as the type of
$arg. If the type of $arg is a type derived from one of the numeric types,
the result is an instance of the base numeric type.

The first signature of this function produces the same result as the second
signature with $precision=0.

For arguments of type xs:float and xs:double, if the argument is NaN,
positive or negative zero, or positive or negative infinity, then the result
is the same as the argument. In all other cases, the argument is cast to
xs:decimal, the function is applied to this xs:decimal value, and the
resulting xs:decimal value is cast back to xs:float or xs:double as
appropriate to form the function result. If the resulting xs:decimal value
is zero, then
positive or negative zero is returned according to the sign of the original
argument. 

Note that the process of casting to xs:decimal may result in an error
[FOCA0001]. 
</proposal>

This produces a well-defined and interoperable result provided that the
implementation supports sufficient decimal range and precision: though not
always the result that a naive user might expect.  For the example in
question

round-half-to-even(xs:float(150.0150e0), 2)

any implementation that supports the required 18 digits for xs:decimal will
convert the argument to the xs:decimal

150.014999389....

which will then be rounded to the xs:decimal

150.01

which will be converted back to the xs:float whose exact value is

150.0099945068.....

which will typically be displayed as

150.01

Michael Kay
http://www.saxonica.com/

Received on Tuesday, 24 October 2006 15:50:22 UTC