"Internal Lot Links" mess in Accounts Receivable

[David Beattie]

Geert,

Thank you of course for your work in development.  I am a former developer myself, so I know how time consuming and tricky it can be to get things right, so don't let my late-coming critical feedback give you the impression that your work is unappreciated!  I hope that my thoughts will be found to be helpful rather than a distraction, but if what I have to say just ends up showing an ignorance of the internal workings of GnuCash, then... well you can just take it as the perspective of somebody trying to figure out how to use the new features, and finding them to be a work in progress... which we already knew.  Version 2.6.0 seems stable, which is the most important thing, and the new features do work, even if there's room for improvement.  And I do like the new features. :)

I have thought a lot about the "lot links" that you implemented, and I understand your design now.  The old GnuCash 2.4 invoices are showing up in GnuCash 2.6 as a single lot for each invoice (perhaps this was the way it was implemented in 2.4 already?), with payments added to the lot until the lot is closed, at which point the invoice is considered paid.  But when payments go to more than one invoice, then the payment transactions had to be split in the register.  By introducing a "lot link", you now avoid creating splits for a payment transaction, and rather enter it in a "pure" manner... and then use a "payment lot" to link this to the "lot link" transaction, where the splits could be split out to the individual invoices with their invididual "invoice lots", and if a payment lot has a positive balance, then that's a pre-payment.  So your new implementation proxies the split assignment through the payment lot and into the lot link transaction, so
 in a sense it's cleaner because a payment itself never has to be split in AR.  But in another sense it's more complicated because it introduces persistent payment lots and an extra "lot link" transaction per payment, and pre-payment and split payments are really only visible in the dialog boxes and can't be seen easily in looking at the raw AR data (if you ignore the Lot Link transactions).  And you have to be careful to think about what might happen if the payment amount is adjusted in the register now, because it's not going to touch the lot links automatically, so you end up with a prepayment or underpayment in the payment lot, and need to re-assign the payment again.  Also, contrary to your simplified statement that you can see what payments pay each invoice in the lot viewer, because the link between payments and invoices is lots.... it's actually not anymore,... the link between payments and invoices is now "lot links", and payments and
 invoices never meet in a single lot anymore, except in the legacy data,... so you can't see this data in the lot viewer for new invoices because it doesn't show the contents of "linked lots" in a special way.

To handle credit notes, it seems obvious that something like a lot link is helpful, since the credit from one invoice can be applied as payment to another invoice, but I'm sure this could have been handled in the simpler 2.4 style as a simple, differential-valued payment (possibly net-zero-valued if the credit matched the invoice amount), that "pays" the positive and negative invoices at the same time with opposite-signed splits--this is the same as a lot link, except it doesn't actually matter if the transaction type is marked as "payment", or "lot link", as long as the positive and negative invoice splits close out the invoice lots in question so they can be marked paid.


So I still think it makes more sense the simpler way, to have invoice lots only, and be much more sparing with payment lots, but that's my opinion and I'm not the one writing the code, so I may be wrong.  In the old 2.4 implementation, payments were split and assigned automatically, and sometimes wrongly, so adding a dialog box to do the explicit assignment is a big improvement!  I imagine the new dialog could have just tweaked the splits for the payment, to make as many payment splits as assigned invoices, and added each payment split directly to the proper invoice lot; any leftover pre-payment split would have to be assigned to a temporary "pre-payment lot", which would disappear once the pre-payment were fully assigned to pay a future invoice.  Of course your implementation went a different direction, always making a permanent "payment lot" for each payment, and splitting the payment via the "lot link" transaction instead, and of course this will
 work if it's carefully implemented, I just have to wonder if it's unnecessarily complicated.  I read your list of problems you were trying to address with the new implementation, and none of them seem to require the lot links to me, at my admittedly naive distance from the source code.  If you still say they're necessary, I'll have to trust your judgment.  But I think reconciliation will continue to be a problem, perhaps worse now than before.  It definitely would have helped usability in GnuCash 2.4 if reconciliation flags were removed on payment splits that were paying an invoice, when that invoice was unposted.  But really, who is reconciling the Accounts Receivable?  (I was, obviously, since I actually did run into this as a problem.)  You already said we shouldn't be poking around in the AR register anyway!  With lot links present, reconciliation in Accounts Receivable is UGLY now and takes a lot of thought, so I probably won't even try it
 anymore.  But this was never necessary, either in 2.4 or now in 2.6, because we have intelligent reports like "Receivable Aging" to filter and display the Accounts Receivable data for us.  Rather, it is the asset accounts that payments are deposited in, that need to be reconciled, and this is unaffected by unposting an invoice that a payment may be assigned to, since it's the split in the asset account that's marked reconciled in that usage scenario, not the split in Accounts Receivable.  Reconciled splits from invoice line items can cause problems once re-posted, but that's an unrelated issue.  Similarly, deleting a payment may cause reconciliation problems in the asset account if that's already reconciled, but that's just asking for trouble!, and that should be covered by some sort of "delete reconciled transaction?" confirmation box, and that's also unrelated to unposting the invoice, except that we lose the assignment of the payment when it's
 unposted.

I actually have practical suggestions, believe it or not!

The "Assign Payment To", dialog is creating the Lot Link transactions, and I was wrong in stating that I couldn't make it work.  The reason it was coming up empty for me is that it doesn't auto-populate the Customer that the payment originally was intended for.  If there is a record of the customer number in the payment transaction (aside from the obvious, the memo text), then populating it automatically into the Assign Payment dialog would go a long way to making this easier to use.  The user could always change the customer in the dialog box if he needed to, although if so, I wonder if a dialog box confirming or ignoring an autocorrect to change the memo field on the payment to the new customer name would be helpful.  If the customer is no longer referred to in the payment transaction, then a non-optional pop-up to choose the customer would be the obvious solution, otherwise it takes a while to figure out what went wrong for a new user like me.

Another helpful change to the current "Assign Payment To"... feature would be to show any existing payment assignment in the dialog.  Currently, the Assign Payment feature can be run twice in a row, and the second time you run it, the stuff you've already assigned is not there, and there's also no indication that the payment is fully assigned already (the amount is still there in full).  I'm scared to find out what would happen to the original payment assignment if I did in fact try to assign it to another, unpaid invoice, and I don't want to corrupt my live data to risk finding out.  But I would suggest that populating that dialog with the UNION of the currently-assigned invoice splits and the available, unpaid invoice amounts, would be more user-friendly than what it currently does.  Then if a payment got mis-assigned, the assignment could be edited in an obvious way.  Doing the re-assignment could even change the amount sent to an invoice that
 the payment was already partly assigned to, in which case, the new amount could be combined in with the existing lot link, rather than adding a second split to the same invoice in the lot link.  Thus the "assign payment" would essentially be an edit button for the lot link, rather than only designed to create them.  Also, the "Assign Payment" menu item could be removed from Invoice lines (except credit notes) in the register, since it doesn't make sense there.  AND, it doesn't make sense to assign the payment to itself, so if assigning an unassigned payment, the payment itself should be deleted from the assignment dialog (it is visible there, currently, listed as a pre-payment).

Yet more feedback on the "Assign Payment": every time I assign a payment, I'm getting an "Imbalance-USD" split, with no value, added to the payment transaction.  I can delete it with no negative consequences.  This brings to mind the idea of what would happen if a payment amount was edited though... I am imagining in the future, an option to trigger the Assign Payment dialog box to come up automatically if a payment amount is ever edited in the register.

I do think it's a bit naive to expect us, real users of the program, to never have to go poking around in the internals of Accounts Receivable.  Until lot links are automatically deleted when payments are deleted (they obviously should be!), then we'll have to go in there to manually delete them, or correct their dates to make sense.  And in terms of the actual payments, I should be able to edit them either from their asset account, or from the Accounts Receivable account, just like any other transaction can be edited from any of the accounts its splits belong to.  Most importantly for the new features, where else would I go if I wanted to "Assign as Payment"?   Don't I have to do that from the Accounts Receivable register?  So if Lot Links are going to remain as the internal implementation of payment assignments, and if they can be fully maintained by the Assign as Payment dialog box, then I think there will soon be no reason to show them in the
 register (once the bugs are worked out), so why not hide them?  That way when we do have to go into Accounts Receivable to re-assign a payment, we won't have to see them, while we will, in fact, be editing them behind the scenes.

Also, whether visible or not, assuming Lot Links are here to stay, I think it would be helpful to anybody who happened to poke around in the "View Lots" dialog, if the date of the Lot Link transaction would be synchronized with the date of the actual payment, just in case the payment date is adjusted.  Otherwise you get some strange artifacts with both the payment and invoice lots being "open" for potentially very long times if you enter an incorrect payment date and then fix it.  And it would have the added benefit of being able to easily see, for any invoice (perhaps shown in a report which would read the data from the invoice lot), the date on which an invoice was fully paid, which could be correct even when the payment date had been adjusted after entering it.

Again, thanks so much for your hard work,

Best regards,

David Beattie


On Monday, January 13, 2014 2:17 PM, Geert Janssens <janssens-geert at telenet.be> wrote:
 
Hi David,
 
Thank you for your feedback and sorry for the bad experience. But as you already mention yourself, it would have helped a lot if you gave this feedback during the development cycle.
 
Anyway further comments in between:
 
On Monday 13 January 2014 13:19:43 David Beattie wrote:
> Hi,
> 
> I'm new to this e-mail list, and I'm not sure if this discussion
> better belongs in the -devel e-mail list.  If so, I apologize, and
> somebody please let me know and I can ask it again over there, but my
> best guess was to ask it here.
> 
> I am a long-time user of GnuCash for business purposes, since the 2.2
> versions.  I just upgraded to 2.6.0 as soon as it was released.
>  Overall I am liking the new business features in concept, especially
> things like credit notes, and explicitly marking payments as applying
> to multiple specific invoices.  Unfortunately some of what I'm about
> to mention is probably feedback I should gave given during the "beta"
> testing phase (2.5 releases), but, since using GnuCash is critical
> for my business, I didn't want to risk data corruption, so I waited
> until 2.6 to even try it.
> 
That's fair enough. Let's work from where we are now.
 
> One thing that I think is unsatisfactory, is the way the Accounts
> Receivable works now.  Probably because of the new support for credit
> notes and the new explicit multiple-invoice payment feature I like,
> EVERY posted payment now incurs a "Lot Link" transaction in the
> Accounts Receivable register.  The problem with this is several fold:
> 
> 1) It clutters the Accounts Receivable register.  Rather than just two
> items per invoice (the invoice itself and the payment), there are now
> 4 showing, since the "lot link" takes up 2 lines on the screen,...
> and even worse, the lot links are not visibly associated with any
> payment, except by date, so it sometimes takes extra thought and
> guesswork to figure out what they belong to if this ever becomes
> necessary.  If they had a reference number of the invoice, or
> check/payment number, that they were associated with, at least it
> would be definitively clear which invoice or check they belonged to.
The Accounts Receivable register was never meant for direct interaction. So the idea that the additional transactions would "clutter up" this register was never considered and probably won't be in the future either.
 
> 2) The only association of a "Lot Link" transaction to the actual
> invoice and payment seems to be the date, BUT, if I enter a payment
> with the wrong date, then the "Lot Link" entries are given the date I
> mistakenly gave to the payment.  This means if I change the date on
> the payment, but not on the Lot Link entries, then the Lot Link
> entries are now in a random place in the timeline, and the only way
> to track them down is by the transaction amount, assuming that even
> matches.
Actually as the transaction name suggests the association between the invoice and the payment is a lot, not a date. So if you want to know which invoice is paid by which payments you can use the lot viewer which you find under "Actions->View Lots..."
I'll admit the interface probably can use some improvements though.
 
> 3) Deleting payments no longer works (at least, not easily).
>  Sometimes I make a mistake when entering a payment.  While I can
> imagine a dialog-based user interface to take care of this issue,
> none exists in GnuCash, so my old solution was to simply find the
> errant payment in the Accounts Receivable register, or in the asset
> account that the payment was transferred to, and delete it.  Now, I
> have to not only delete the payment, but the lot link transaction.
>  If I don't delete the lot link transaction, then the invoice is
> still marked as paid, which is WRONG, since the payment no longer
> exists in Accounts Receivable.  If, on the other hand, I delete the
> lot link transaction only, and not the actual payment, then it is no
> longer marked as paid, which is also wrong, since the payment is
> still listed and the accounts receivable would still reconcile for
> that customer.
This is clearly a bug indeed. If a payment is deleted, the invoice should no longer be marked as paid.
 
Until a fix has been released you may be able to avoid this by not deleting the payment, but instead modify it to the correct state. Granted that is not possible for all types of corrections.
 
> 4) Un-posting and re-posting invoices no longer works,
> in any way.  If I process payment for an invoice, then later need to
> modify the invoice, to add to it, for example (which would render it
> only partially paid, with an amount due), or even to edit details
> such as the account a line item was associated with, then re-post the
> invoice, the older versions of GnuCash would automatically
> re-associate the payment with the invoice, and it would automatically
> be marked as paid (or partially paid, depending on the
> circumstances).  In 2.6.0, this no longer works.  The Lot Link
> transaction goes away when the invoice is un-posted.
This is deliberate.
 
The code in 2.4 assumes that when you unpost an invoice you will eventually repost it again with minor changes (a different date, amounts changed,...). But it couldn't cope properly with other use cases like
- discard the invoice and recycle it for another customer
- when the total invoice amount decreased to be less than the already applied payment, this could mess up the accounts receivable register
- if the payment was already reconciled unposting could mess up reconciliation
- you could decide to post to another account, in which case the payment becomes invalid for that invoice.
So there were lots of corner cases that could cause trouble when the unposted invoice remained linked to the payment.
 
In 2.6 the underlying idea is that gnucash can't know what you will do with the invoice after you have unposted it. On the other hand your payments may have been reconciled or at least explicitly entered. So when an invoice is unposted the only sensible thing to do is to drop the link between the invoice and the payment and keep the payment around.
 
>  There is an
> "Assign As Payment" menu item for each transaction, but I've never
> successfully made this work yet for any purpose.  In this
> circumstance, it brings up an empty dialog box with no invoices
> available to be paid, despite the fact that the invoice is NOT marked
> as paid anymore.
That's probably a bug as well. I will check that later.
 
>  The normal/new "process payment" dialog box shows
> the previous payment available as a "pre-payment", and shows the
> invoice needing to be paid, but even trying to enter a $0.00
> psuedo-payment to associate the two fails, because a payment of $0.00
> is not accepted.
Hmm, I think that was supposed to work... That may have broken again then at some point. Should be fixed.
 
>  I have to assume that the "pre-payment"
> functionality of 2.4 and earlier GnuCash is completely broken now?
>  The only solution I can find is to write the payment details on a
> scrap of paper, delete it, and re-enter it after the invoice is
> re-posted.
 
> 5) Clearly these lot links are still optional internally,
> because all my old invoices before the 2.6 upgrade are still marked
> as paid, correctly, without them.  So I have old data that is not
> cluttered, and new data that is cluttered, but no way to enter data
> in the old, uncluttered way, since the "lot links" are automatically
> generated and the new payments don't work without them.
> 
They are not optional. They are the new way of doing payments. The code can still handle the old style because that is a simplified/restricted subset of the logic. But the old style can't handle credit notes nor the more flexible payment assignments.
 
> 6) The comment on the splits says "Internal link between invoice and
> payment lots".  In my opinion as a user, "Internal" means I shouldn't
> have to see it.  So I was looking for a way to hide them, but there
> is no filter to filter out transactions by type, so I have to see the
> "Internal" data, which I would rather hide except when briefly
> necessary in order to de-tangle some problem (like when deleting
> payments).  This bothers me philosophically.
As I mentioned before, the whole Accounts Receivable type of account is not meant for human consumption. You should rarely interact with it directly. You should be able to do all business related actions via other ways. If something can't be done, that needs to be improved.
> 
> Is there a plan to eliminate this mess in a future release?  I hope
> so!  For now I can live with it, it just makes correcting minor
> mistakes, or changing anything about an invoice or a payment after
> the fact, a BIG pain.  In my opinion, the Lot Links should be
> re-thought completely, and gotten rid of, and the concept
> re-engineered in a simpler way that doesn't have all the problems I
> mentioned above.  I'm tempted to go back to version 2.4.15 until this
> is addressed, but not sure if the data file would be backwards
> compatible so I'm scared to downgrade.
I don't think lot links will go away. They're the only reasonable concept I could come up with to cater for credit notes and more flexible payment assignments *and* at the same time stick to a format that 2.4 won't choke upon.
 
A few bugs should clearly be fixed and a few cases need more user friendly interaction that I can agree to.
 
If you have created a credit note, you can't go back to 2.4. It will simply refuse to open your data file. Other the lot links may introduce some distractions in your reports, but by my knowledge won't break them.
 
Geert