[Bug 2989] Datatypes 2006-02-17 WD: floatingPointRound function

http://www.w3.org/Bugs/Public/show_bug.cgi?id=2989

           Summary: Datatypes 2006-02-17 WD: floatingPointRound function
           Product: XML Schema
           Version: 1.1 only
          Platform: Macintosh
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Datatypes: XSD Part 2
        AssignedTo: cmsmcq@w3.org
        ReportedBy: xan.gregg@jmp.com
         QAContact: www-xml-schema-comments@w3.org


The function floatingPointRound() in APpendix E.1 appears to have incorrect
and/or redundant information. Below is an excerpt, converted to ASCII.

   2. So select e that 2^cWidth * 2^(e ? 1) < |nV| <= 2^cWidth * 2^e .
   3. So select c that  (c ? 1) * 2^e <= |nV| <c × 2^e  and  2^(cWidth?1) < c
<= 2^cWidth .

It's the second clause of step #3 (3b) that is at issue (the part after "and"). 

"2^(cWidth?1) < c" seems redundant since is c <= 2^(cWidth-1), then by the
(3a), |nV| < 2^(cWidth -1 + e), but by (2), we already have 2^(cWidth + e - 1)
< |nV|.

"c <= 2^cWidth" indicates something is wrong. Suppose |nV| == 2^cWidth * 2^e,
as permitted by (2). Then,  |nV| < c * 2^e (from (3a)) implies 2^cWidth * 2^e <
c * 2^e which reduces to 2^cWidth < c, so c can't be less than or equal to
2^cWidth as (3b) requires.

Perhaps the ... <= |nV| < ... in (3a) should be ... < |nV| <= ... and (3b) is
entirely redundant.

Received on Tuesday, 7 March 2006 04:18:11 UTC