How do we handle --

Richard Wackerbarth rkw@dataplex.net
Fri, 28 Jul 2000 17:15:19 -0500

On Fri, 28 Jul 2000, Jason Rennie wrote:
> So, here something from left field...
>
> All of this discussion concerning roundoff error, internal number
> representation, etc. seems to be assuming that there *is* a right answer.
> i.e. if we discuss and think hard enough, we'll be able to come up with a
> solution that will always do the right think with respect to roundoff
> error.
>
> Another approach would be to assume that we can't win, go with the
> floating point representation that we have and allow the user to fix
> roundoff "errors" by explicitly stating what a*b equals when a
> price*amount calculation doesn't match what it needs to be.

Actually, there are two problems.
Here I agree with your suggestion. In effect, we "guess" at an answer
and allow the user to correct it if we are wrong.
2) Precision of calculation.
This is a numerical analysis problem. Even if we display the correct
result for each entry, we will eventually end up with a "roundoff" error
whereby the total is not equal to the sum of the displayed parts. For
accounting purposes, this is not acceptable. As a result, we must be able
to represent the exact number displayed, not just an approximation to it.
We do this by counting in units of the smallest displayed change. (For
example cents rather than dollars). Thus the sum will be exact and roundoff
will not accumulate. After the calculation, we can convert to any desired
units for display. (Display 123 cents as \$1.23)

Some may have forgotten that it is not floating point, per se, that is the
problem. It is that floating point dollars cannot form an exact
representation of cents. It is also a problem that we must adjust each
displayed quantity to be exactly the number displayed. Although this can be
done in floating point, it is much easier to do so in integer. Also, since
you don't waste bits for an exponent, you can represent a larger value within
the same register size.