2.6.9 exchange rate structure change and reports

[John Ralls]

> On Oct 22, 2015, at 4:22 PM, Wm... <tcnw81 at tarrcity.demon.co.uk> wrote:
> 
>> Thu, 22 Oct 2015 10:19:53 <5278634.zf4xtApTfp at legolas.kobaltwit.lan>
>> Geert Janssens <geert.gnucash at kobaltwit.be> wrote...
>> 
>>> On Wednesday 21 October 2015 19:29:49 John Ralls wrote:
>>>> > On Oct 20, 2015, at 3:17 PM, Wm... <tcnw81 at tarrcity.demon.co.uk>
>>>> > wrote:
>>>> >
>>>> > Fri, 16 Oct 2015 07:20:45
>>>> > <380C9B08-7C86-49C3-879E-4115216C7575 at ceridwen.us> John Ralls
>>>> > <jralls at ceridwen.us> wrote...
>>>> >
>>>> >>> On Oct 15, 2015, at 4:04 PM, Wm... <tcnw81 at tarrcity.demon.co.uk>
>>> >>
>>> >> wrote:
>>> >>> I'm not sure if I'm missing something but should reports, e.g.
>>> >>
>>> >> balance sheets, know about the fact that exchange rates that were
>>> >>
>>> >>> fraction buys 1
>>> >>>
>>> >>> are now
>>> >>>
>>> >>> 1 buys many?
>>> >>
>>> >> My reports seem to be stuck at the exchange rate last collected
>>> >> using
>>> >
>>> > the 2.6.8 structure.
>>> >
>>> >> Hopefully I'm doing something wrong otherwise all reports involving
>>> >
>>> > currencies that inverted are going to be increasingly out of date
>>> > over time.>
>>> >> An example of not-insignificant currencies inverting is GBP / USD
>>> >
>>> > Yes, and I had thought that they already checked in both directions.
>>> > I’ll take a look.
>>> >
>>> >
>>> >
>>> > The wrong exchange rate is being used on the main accounts display
>>> > too if one of the "Name (Currency)" columns is displayed.
>>> >
>>> > I've tried having a look in the code but it is taking me too long to
>>> > find.
>>> >
>>> > If there are no pre 2.6.9 prices in a book it gets the right one.
>>> >
>>> > Not sure if this is a re-opening of an old bug but I have added a
>>> > new one with example books anyway and hope I got it right
>>> >
>>> > https://bugzilla.gnome.org/show_bug.cgi?id=756889
>>> 
>>> Yeah, it only looks at the “reverse” quotes if there are no “forward”
>>> quotes, and even that’s in only a few places.
>> 
>> That's a change I once made while working on the transfer dialog. Before it didn't even
>> consider "reverse" quotes. With your work on the pricedb this would have to be updated again
>> to look for a price in both forward and reverse quotes and prefer the one closest in time.
>> 
>>> I don’t know why I
>>> didn’t recognize that that was the wrong way when I first looked at
>>> it. A possible work-around if you’re not interested in historical
>>> prices is to use the “Remove Old” button on the price editor. That
>>> won’t fix everything, but it will fix the places where it looks for
>>> “reverse” quotes and where the direction doesn’t switch much.
>> 
> I can work around that but I think it will make many people nervous. For e.g. you'd need to tell them whether or not their existing transactions would change or not.  I think someone else mentioned this up thread.
>> 
>>> I’ve got a better design in progress, where both directions are
>>> queried and the results sorted chronologically. It’s in the pricedb
>>> code so there will be no issue of having to call the query in each
>>> direction as is the case now, but calling code will need to check the
>>> direction of each returned price and use it accordingly.
>>> 
>> Even better.
>> 
> It makes sense to me for the request to be "give me a rate for this pair of currencies at this date" and for the pricedb to be responsible for the return.
> 
> Obs "me crap at reading code" as much as anything: as mentioned above I got lost going through the code trying to work out where the entry and exit points were.  I could see the data clearly enough but there seems to be a heck of a lot of stuff that is just mesh.

That’s unfortunately true of a lot of GnuCash code, and one of the motivations for the C++ rewrite.

> 
> Surely the pricedb is a resource to other bits of gnc?
> 
> the price of an actual exchange rate transaction is fixed in that transaction, the pricedb is _just_ used for valuing.
> 
> in extremis, the whole pricedb could be trashed and the value (of various things at the last time they were exchanged) would still be intact.
> 
> Or have I got that wrong?

That’s exactly right. The pricedb is used only for valuing assets on the Accounts page and in various reports and for *proposing* a price in the Transfer Dialog. Splits don’t even contain a price, just an amount in their own account’s commodity and a value in the transaction’s commodity. The ratio of amount/value is the imputed price.

> 
>>> This is becoming a much bigger change than I initially thought it
>>> would be. Perhaps it would be better to revert the pricedb back to
>>> the 2.6.7 state and apply it to master for the next major version.
>>> Any votes one way or the other?
>>> 
>> I agree it's getting big for the stable series. So agreed to revert to the 2.6.7 state and do the
>> work on master only.
>> 
>> Will reverting back have any side effects considering some users have already been fetching
>> quotes with the new system ? I guess that some will need to manually enter prices in the
>> opposite direction if they want to have historical quotes for the 2.6.8/2.6.9 period ? On the
>> other hand they'll have to do that now already as well and will continue to have to do so if we
>> don't revert/until your better design is implemented. Something to mention in the release
>> notes, I guess...
>> 
> 
> I think I am with all in saying revert to 2.6.7 until we can do this well (I am positive about this change in general and for the future)
> 
> Argument:
> 
> the short term *need* for the 2.6.[89] structure change is probably cosmetic to most users, i.e. there are relatively few users with disappearing decimal exchange rate problems
> 
> the longer the current situation continues the more (increasing number of) people are going to find their reports and assumptions about exchange rates are unexpected even if they don't have a remote need for the problem being solved (too few decimals to care <= most of world plus dog)
> 
> at the moment the data recording post 2.6.7 is understood and can be repaired [1], if there are other changes in the future then it could become very difficult to work back from that point in time
> 
> [1] with a text editor, an understanding of XML and a calculator, for example
> 
> although I lazily use gnc's pricedb for checking exchange rates I know I shouldn't and have argued publicly that other people should not depend on them.  my ongoing solution (I'm in love with pandas [snakes not bears] at the moment) handles missing data rather well.
> 
> [pandas] http://pandas.pydata.org/

The actual motivation for the change was some users’ frustration when entering several multi-currency transactions at once: Each transaction created a new price which the Transfer Dialog dutifully presented as the price for the next transaction. Thanks to rounding of the value that price wasn’t necessarily the price the user had entered, nor was it necessarily representable as a decimal number. The result was that the user had to keep re-entering the proper exchange rate for every transaction, which is pretty annoying.

I originally proposed rounding the exchange rates, and Mike Alexander pointed out that that was likely to cause trouble where the exchange rate was large (or small, depending on direction). The solution I eventually settled upon had three parts: Limiting the number of stored prices to one per day, prioritizing the sources so that prices from some sources would be preferred when deciding whether to overwrite a price already in the pricedb, and (almost) always storing the price in the direction that’s > 1 when both commodities are currencies.

You raise a good point, that only the last bit is causing problems. Having reverted the whole mess yesterday, I suppose I should try to separate out the first two parts, single price per day and source preference, and reintroduce them to maint. Those two will, I think, deal with most of the user frustration issues and leave the rounding problems for master, where I’ve already done a fair amount of work to improve our handling of large/small exchange rates and larger denominators to support Bitcoin for the next major version.

Regards,
John Ralls