From: Michael Kay <mhk@mhk.me.uk>
Date: Mon, 5 Apr 2004 17:36:02 +0100
To: "'Jeni Tennison'" <jeni@jenitennison.com>

Message-Id: <20040405163628.72AA7A2F61@frink.w3.org>

> >
> > If the types of \$arg1 and \$arg2 are xs:integer or xs:decimal, then
> > the result is of type xs:decimal. The precision of the result is
> > implementation-defined, but it must be not be less than min((18,
> > max((p1, p2)), R)) digits, where p1 is the precision of \$arg1, p2 is
> > the precision of \$arg2, and R is the number of digits (possibly
> > infinite) required to represent the exact mathematical result.
> > "Precision" here means the total number of significant decimal
> > digits in the value; all digits are considered significant other
> > than leading zeros before the decimal point and trailing zeros after
> > the decimal point. If rounding is necessary, the value must be
> > rounded towards zero. Handling of overflow and underflow is defined
> > in section 6.2.
>
> Using the precision of the two arguments to determine the precision of
> the result leads to results that I find strange. For example:
>
>   1       div 3         =>  0.3
>   1000000 div 3000000   =>  0.3333333
>   0.00001 div 0.00003   =>  0.33333
>
> It would make a lot more sense to me to just use min((18, R)), but any
> definition here is better than none. What cases were you thinking of
> that led you to suggest using the precision of the arguments to
> determine the precision of the result?
>
You're right, the formula doesn't work in these cases. The cases I was
thinking of are where p1 or p2 are >18, which would mean the correct formula
is

min((max((18, p1, p2)), R))

Michael Kay
Received on Monday, 5 April 2004 12:36:28 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 16:56:56 UTC