[Bug 3528] additional expected values for decimal operations, part 2

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3528





------- Comment #1 from mike@saxonica.com  2006-07-24 17:57 -------
Concerning extvardeclwithouttype-18, this effectively returns the result of (7
div 3) (which is an xs:decimal). This is governed by the rule:

For xs:decimal values the number of digits of precision returned by the numeric
operators is ·implementation-defined·. If the number of digits in the result
exceeds the number of digits that the implementation supports, the result is
truncated or rounded in an ·implementation-defined· manner.

Implementations are required to support at least 18 digits. My reading is that
you have to take the two sentences together, and this means that the result of
the division must be at least 18 digits. Therefore the result 2.333333333333
is wrong. 

On a more practical point, however, it's clear that there are an infinite
number of correct results for this query. The simplest solution would therefore
be to change the query so that instead of computing avg(4,2,1), it computes
avg(8,4,2,1), which is incontrovertibly 7.5.

Concerning extvardeclwithtype-18, the result here is the decimal value of (8
div 3) converted to a float. Similar considerations therefore apply. 

Concerning fn-avg-mix-args-009 teh query here is:

fn:avg( ( ( xs:decimal("-1.000000000001"), xs:integer("-100"))))

The correct answer to this can be expressed exactly in 15 decimal digits of
precision, and since all products are required to support this number of
digits, I do not think any answer other than -50.5000000000005 is acceptable.

Received on Monday, 24 July 2006 17:58:01 UTC