[Bug 29409] New: [FO31] fn:format-number with xs:double/float and (non-)rounding

https://www.w3.org/Bugs/Public/show_bug.cgi?id=29409

            Bug ID: 29409
           Summary: [FO31] fn:format-number with xs:double/float and
                    (non-)rounding
           Product: XPath / XQuery / XSLT
           Version: Candidate Recommendation
          Hardware: PC
                OS: Windows NT
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Functions and Operators 3.1
          Assignee: mike@saxonica.com
          Reporter: abel.braaksma@xs4all.nl
        QA Contact: public-qt-comments@w3.org
  Target Milestone: ---

Section 4.7.5 Formatting the Number says, under item 6:

"The mantissa is converted (if necessary) to an xs:decimal value, using an
implementation of xs:decimal that imposes no limits on the totalDigits or
fractionDigits facets. "

And, on xs:double/float specifically:

"the one that is chosen should be one with the smallest possible number of
digits not counting leading or trailing zeroes (whether significant or
insignificant). For example, 1.0 is preferred to 0.9999999999, and 100000000 is
preferred to 100000001" 

I fail to see how 100000000 is preferred over 100000001 (both have the same
amount of digits, the second having in part non-zero trailing digits). I also
don't see how this is supposed to work. Consider:

format-number(1E26, '#')

This can return '10000000000000000905969664' because converting the double
value 1E26 to xs:decimal is 10000000000000000905969664.

However, the text may imply that this should in fact return
'10000000000000000000000000', but I don't see how that is supposed to happen.

The way I understand the text is that the example above should format the same
way as format-number(xs:decimal(1E26), '#'), where there's no limit in the
decimal digits. This seems in line with other parts of the spec where
conversion from double to decimal retains the insignificant digits.

Relevant tests that show a different interpretation are numberformat60a,
numberformat60m, numberformat60n, numberformat60o, numberformat60p,
numberformat60q. Taking the latter:

format(1E100 div 3, '#####################')

I think this should (may, ought to, must?) return 
'3333333333333333224453896013722304246165110619355184909726539264904319486405759542029132894851563520', 

and not 
'3333333333333333000000000000000000000000000000000000000000000000000000000000000000000000000000000000'

But apart from the relevant tests, I raised this as a spec bug because, even if
the text is considered correct, perhaps an example could be given for this
particular entry.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

Received on Sunday, 31 January 2016 02:24:23 UTC