[Bug 2989] Datatypes 2006-02-17 WD: floatingPointRound function

http://www.w3.org/Bugs/Public/show_bug.cgi?id=2989





------- Comment #1 from davep@iit.edu  2006-03-09 03:31 -------
(In reply to comment #0)
> Perhaps the ... <= |nV| < ... in (3a) should be ... < |nV| <= ... and (3b) is
> entirely redundant.

Actually, it's 2 that is in error.  The '<' and '<=' should be switched.  That
puts |nV| in the proper top-open interval of normaiized values corresponding to
e.  Then 3 correctly finds adjacent values (c-1)*2^e and c*2^e in the *closed*
interval that bracket |nV|.  If |nV| *is* one of the floating point values, the
'<= will insure that we ultimately get the one that is in the interval.

(Note that in 3b. the '<' and '<=' should *not* be switched.  The '<' insures
that (c-1)*2^e is in the e-segment and the '<=' allows the approximation to
round correctly for exact values higher than the last floating-point value in
the partition.)

It's entirely possible that an exact value |nV| very close to the top of the
e-segment will round to the floating-point value that is the top of the segment
and is not part of the interval, but at that point the algorithm no longer
cares what segment the rounded-to value is in.

Of course, I'm fallible; check me on this!

Received on Thursday, 9 March 2006 03:31:18 UTC