Rethinking Numeric

[John Ralls]

On May 31, 2014, at 4:24 PM, John Ralls <jralls at ceridwen.us> wrote:

> 
> On May 31, 2014, at 3:20 PM, Mike Alexander <mta at umich.edu> wrote:
> 
>> --On May 31, 2014 11:19:23 PM +0200 Christian Stimming <christian at cstimming.de> wrote:
>> 
>>> But back to your initial question: You said we occasionally
>>> "encounter  overflow errors". I don't understand (yet) what the
>>> actual problem is. With  our current rational numbers and int64_t
>>> numerator we have approx. 19 decimal  digits of precision (see [2]
>>> for the digits of a 64 bit signed integer), if I  consider the
>>> numerator as fully used.
>>> 
>>> Are 19 significant decimal digits not enough? Are there thinkable
>>> cases when  they are not enough? I tend to think the problem is
>>> rather found in our  rational number's rounding, which is not the
>>> suitable rounding method for our  financial application domain. If
>>> this is the problem, a different data type  that does the rounding
>>> always according to decimal numbers, and not according  to (in normal
>>> float/double calculations) binary floating point numbers, or (in
>>> gnc_numeric) according to rational numbers with some potentially
>>> unknown  denominator.
>>> 
>> 
>> Rounding along with division is at least part of the cause of overflows.  I dealt with a number of overflow problems in the advanced portfolio report that were the result of the interaction of rounding and division. I solved it by essentially converting things to a decimal notation.  I picked a (hopefully) sufficiently large decimal denominator and rounded relevant calculations to that denominator.  I think this is more or less what you're suggesting in the context of the current implementation.
>> 
>> Division of rational numbers, which is what GnuCash does, involves the calculation of GCDs (or something similar) which can produce very large integers.  Years ago I was involved with the Reduce algebraic system which implements arbitrary precision rational numbers mainly for this reason.  It needs to exactly divide rational numbers with no rounding so it can, for example, factor polynomials.  All but the most trivial calculations quickly exceeded the range of hardware integers.  We don't need that, of course.  Our problem seems to be that we're using rational numbers where it isn't really necessary.
> 
> Yes, this is indeed it, and my original proposal was to use a multi-precision library wrapped in a rational number library, and to convert the results to fixed-point decimal with a lot of digits to the right of the decimal point, hoping that that would cover the rounding issue.
> 
> So I’ll do a trial implementation and put it on my GitHub repo for everyone to test out. It looks like there are four FOSS libraries available to choose from: The afore-mentioned bytereef library, ICU’s decNumber [1], GCC’s libdecnumber, and GCC’s std::decimal in libstdc++. The last is an implementation of a TR for C++ [2], [3]. libdecnumber apparently requires enabling at compile time and neither it nor std::decimal work with Clang. I’ve already downloaded and built the bytereef library so I’ll go with that rather than ICU decNumber. There are a few more bugs on my list to take care of, so it’ll be next week some time.

I’ve got the start of the trial implementation pushed to https://github.com/jralls/gnucash/tree/libmpdecimal. It builds, but it doesn’t pass all of the tests yet. I think it’s quite promising and I’ll keep working on it, but I wanted to give everyone a chance to see what it looks like. So far I’m leaving the existing API in place to simplify getting it working, but you’ll notice that the guts of the implementation is in a C++ wrapper to make it easy to use directly as the C++ rewrite progresses.

For those who don’t already know, I’m leaving Thursday for a month's vacation in Brussels and London. I expect I’ll check in here daily and may pop up on IRC from time to time, but I’m not planning to do a lot of GnuCash work.

Regards,
John Ralls