[GNC] Why do Imported Transactions NEED to be Matched?

Sun Apr 26 18:05:13 EDT 2020

Tom

Please Reply to the gnucash-user at gnucash.org (Reply to List or Reply to All) mailing list rather than individual
contributors. The issue will get a wider response and the replies are then useful to all of the list participants.

I am not a major contributor to the development but I think you will find the main reason for having to associate the
external file names with the internal file names is that GnuCash does not use the account name strings internally for
referring to accounts in the code but uses GUIDs as a searches for the GUID is likely more rapid than a search for a
string. Therefore when the importer hits an account name you have entered in the transfer account column, and there is
not a mapping of that string to the GUID, the software currently does not know where to direct it.  A string search
through the whole account structure which has to match a series of sub-strings is likely to be fairly expensive in
computer time which is one possible reason the manual mapping was implemented.  It should be just as efficient to map
the Quicken categories directly onto the GnuCash account structure.

In principle it is obviously possible to have GnuCash search through its list of account data structures for a match to
the entered string and construct that mapping, but that code doesn't exist at the moment. The best way to raise this is
using bugs.gnucash.org and to file a bug as a feature request against the program. This will at least get it on the
developer group's radar. 

After that it is a matter of priorities and someone with the requisite skill sets and knowledge
of the code picking it up and whether it can be implemented without too much of a computing time cost within the current
architecture of GnuCash. The core of GnuCash is quite old (circa 1998) and I think was originally written in a LISP
variant (Scheme) but it is very reliable. There are projects underway to convert the Scheme to C, C++, which fit in
around the fixing of bugs in the existing code, but this has to take place without impact on the core functionality so
progress is not necessarily all that rapid. Unless someone in the development team sees a way to implement this fairly
quickly and easily it may take some considerable time to be up and running. Both the development and documentation teams
are all volunteers and the resources to implement things we would like to see are not always available.

I wouldn't be
holding my breath. It is more productive to come to terms with the limitations in GnuCash and seek workarounds. The CSV
importer was recently completely rewritten and a rewrite of the documentation is underway. The Help manual component in
Ch 6  is largely complete. The Guide component which will address importing strategies and importing multiline
multisplit data is still being written.  Is your data CSV or QIF?

In a given data file, that mapping process should be a one off, but like many things with the importer, AFAIK the map is
probably only written to the data file when you successfully complete the import and if you don't do that or you start a
new data file, it will have no  mapping information stored, which is likely why you end up back at square one.

Are you
working with the full data file or have you pulled out a typical subset, e.g. one that has a transaction to each of the
accounts/categories in the heirarchy, to get the importer setup and the map file constructed?

If you do this first, you should then be able to import larger sections of the files with no problem. When I transferred
from MYOB around 12 years ago I brought it in initally a month at a time and then a year at a time once I had the
process sorted. If you go back through the gnucash-user archives you will find many references to initially splitting up
large imports into small maneagable chunks initially and then only increasing the size of the chunks when you have the
process running adequately. You will also find many discussions around importing Quicken data and optimal strategies.

I
also worked backwards importing the last year's data first, the next to last year's data, resetting the opening balances
to the accounts as I entered each batch of prior data. I still had my old version of MYOB accessible then so I was able
to generate the balance sheets at the start of each imported chunk which helped a lot.

David

On Sun, 2020-04-26 at 20:16 +1000, ts at sgml.net.au wrote:
> David,
> (This is not about matching during import.)
> Over the last six+ weeks I have been importing my Quicken ledger and for various reasons I had to do it over and over
> again.
> Because I purchased Quicken 2007 (with no export facility) I needed to find a way to get all transactions and all
> accounts used for the transactions.
> So I achieved the export of Quicken accounts and Quicken categories and created an accounts list for GC. Then I
> exported all transactions from Quicken (and I had to split it into two exports because Quicken could not give me one
> full export (around 13,000 transactions from 1987 till now).
> I wrote a perl script to convert the Quicken transactions (yes also the splits) into the GC import required format
> (balanced records).
> You know that writing a script does not always succeed the first time, so I had to do this exercise over and over. The
> import of the transactions into a new ledger into GC always had me to manually select the non-linked accounts to the
> already imported accounts. Over the last six or so weeks that drove me nuts.
> Here is my suggestion:
> During import: when the account exists in GC and the import of the transaction refers to that account, then the user
> should not have to manually select the "Not linked account". The import program should have read the existing accounts
> (which it does when reading the XML structure) and automatically assign the account name. Even when there is a case-
> sensitive name change (upper/lower case) but the full name matches, it should honor that name.
> In short: when starting an import of transactions and GC has read the linked accounts, it should honour the non-
> assigned account names and automatically assign these names.
> Looking forward to your response.
> Regards,
> Tom Seesink
> On 2020-04-26 08:29, David Cousens wrote:
> > Mainly because your particular use case is not the only one that the
> > importing process has been designed to meet and the import process has to
> > survive at least some common problems in the implementation by various 
> > sources of files and a considerable variety in the amount of information
> > supplied by different sources of files.
> > 
> > AFAIK you can't skip the initial step of assigning account names in the
> > transfer field to accounts in the GnuCash account heirarchy which is a one
> > off mapping step which occurs the first time an account name appears in a
> > transfer account field. Once that mapping is created it is applied to all
> > future import with that name in the transfer field. There is little value in
> > assigning full account paths here as they still have to be mapped once
> > (although there is no problem in doing so).
> > 
> > If all your data contains a transfer account then you can effectively skip
> > the matching process. If all the backgrounds to all the rows are green and
> > have the A column checked, just hit OK. 
> > A possible new feature here would be an option which could be set in the
> > importer user preferences to automatically trigger the OK if the above
> > criteria were met on all rows. That doesn't exist at the moment.
> > 
> > If any have a gold background a transfer account has not been assigned
> > (could be a typo in the field for example) but this should not happen if you
> > are specifying transfer accounts as the account mapping process should be
> > initiated and you will assign an account.
> > 
> > If either the U+R or R ( recently changed to U+C and C)  checkboxes are
> > checked and the backgound is green then an imported transaction has been
> > matched against an existing transaction. The matching takes place within a
> > date range window and some tolerances on the amount. If you are confident
> > that these have not been imported  previously just change the checkbox to A.
> > 
> > A red background to a row indicates that the transaction did not meet the
> > criteria for automatic assignment and that manual intervention is required
> > to complete import of that row.
> > 
> > David
> > 
> > 
> > 
> > 
> > 
> > -----
> > David Cousens
> > --
> > Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html
> > _______________________________________________
> > gnucash-user mailing list
> > gnucash-user at gnucash.org
> > To update your subscription preferences or to unsubscribe:
> > https://lists.gnucash.org/mailman/listinfo/gnucash-user
> > If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> > -----
> > Please remember to CC this list on all your replies.
> > You can do this by using Reply-To-List or Reply-All.
> 
> -- 
> Thomas A. Seesink
> Arundel QLD 4214 Australia
-- 
Dr David R Cousens
B.Sc, M.Prof. Acc., Ph.D., G.C.Ed