[Bug 29408] [QT3] Precision in extvardeclwithouttype-18

https://www.w3.org/Bugs/Public/show_bug.cgi?id=29408

Michael Kay <mike@saxonica.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mike@saxonica.com

--- Comment #6 from Michael Kay <mike@saxonica.com> ---
F+O says in section 4.2:

For xs:decimal values the number of digits of precision returned by the numeric
operators is ·implementation-defined·.

This is probably far more liberal than we really intended: for example it
allows 1.01 + 1.04 to return 2. However, it doesn't say that the result of
division must have any particular minimum precision, and I don't think we
should read it as if it did.

As a further complication, there is no requirement to implement avg((1,2,4)) as
sum((1,2,4)) div count((1,2,4)): we say that "the implementation may use an
otherwise equivalent algorithm that avoids arithmetic overflow". Avoiding
overflow may well involve loss of precision, and the loss of precision can be
justified by the fact that the precision of "div" is implementation defined. So
a final result like 2.33333333334 is not at all out of the question.

Pragmatically, the spec is loosely worded in this area and it therefore makes
sense for the corresponding tests to have loosely-defined pass criteria.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

Received on Monday, 4 April 2016 08:37:14 UTC