[Bug 3248] Numeric precision

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3248





------- Comment #1 from davep@iit.edu  2006-05-20 02:10 -------
(In reply to comment #0)

> In 3.3.4, it's not clear what the sentence "Precision is sometimes given in
> absolute, sometimes in relative terms" means. Is it referring to the way
> precision is defined in this specification, or is this background information
> about the big bad world? I'm also confused by the definition of "arithmetic
> precision". It has just been stated that precisionDecimal "closely corresponds"
> to a floating point decimal datatype. So what is meant by "[digits] to the
> right of the decimal point"? Is this decimal point a fixed one or a floating
> one?

There are (at least) two ways of expressing "precision":  absolute or
arithmetic
precision (exemplified by "+/- n") and relative or geometric precision
(exemplified
by "+/-n%").  Certain selected values and precisions can have both the
numerical
value and the precision specified with a single numeral (avoiding "+/-").  The
simple technique is to identify the arithmetic precision by the count of digits
to the right of the decimal point, modifying as appropriate when scientific
notation is used.  A so-called "fixed-point" datatype has a value space of
precision-bearing numbers all having the same arithmetic precision.  It is not
feasible to use a single numeral format to describe numbers having the same
geometric precision, but a related concept which might be called "floating-
point" precision can; it is exemplified by limits (either upper bound or fixed)
on the total number of "significant" digits, regardless of the location of the
decimal point in the numeral.  This "floating-point" precision is arithmetic in
the small and geometric in the large.  So-called "floating-point" datatypes fix
or
limit the floating-point precision for all or most of their values with
numerical
"numerical values" (thus excluding NaN and infinities).  The precisionDecimal
datatype does not limit either the arithmetic or floating-point precision of
the precision-bearing values therein, so the question of floating-point or
fixed-point has no bearing.

The minScale and maxScale facets can be used to limit the arithmetic precision;
making the have the same value would yield a "fixed-point" datatype.  The
totalDigits facet can be used to limit the floating-point precision, and if
used
in conjunction with minScale and maxScale can approximate a "floating-point"
datatype analagous to float or double with base-10 rather than binary bases.
(Although it would not allow for rounding lexical representations whose exact
value is not in the value space, which is allowed for float and double.  This
appears less important for base-10-based floating-point datatypes than it
is for binary ones.)

Received on Saturday, 20 May 2006 02:10:29 UTC