[GNC] Fwd: [GNC-dev] Fwd: Failed - import of QIF files from Quicken

[James Peterson]

Well, with no hint of what the problem is, even smaller
test sets won't necessarily find anything, so it could be 
a lot of work for no benefit.  For example, I could export
each account separately (although there are about 80 of them
so that's a lot of work), and each account could load 
okay separately, because maybe the problem is an interaction 
between two different accounts. So then I have to do 80*79
possible pairs of accounts, and so on.

My poking around in the 
source code to try to find the problem suggests that the
problem is in the Scheme part that seems to be working with
all the accounts at the same time:

 ;; Build a local account tree to hold converted transactions.

 ;; Sort the account list on the depth of the account path.

 ;; Make all the accounts.

 ;; Run through the markable transactions marking any
      ;; duplicates.  marked transactions/splits won't get imported.

               ;; Convert into a GnuCash transaction.
               ;; rebalance and commit everything

So this looks like multiple runs thru the data, and even printing
the line number won't narrow it down -- you have to know what
step of the process we are doing, and where we are in the data
at that point.

As things stand, we don't even know what the immediate problem
is -- a divide by zero?  Looking for an account that should exist,
but doesn't?, an account type that it doesn't know about? 

The code is not written to allow it to even explain what it is
doing. If we are going to use this code base, someone who actually
understands, and is comfortable with the code will need to add
a "verbose input processing" level or something that will create
a log file explaining what it is going to try to do, then do it,
then explain what it got and how it interprets that. I've done 
that with code before, but only C code, where, if need be, there is
the equivalent of a printf to a log file between every major step,
or each iteration of a loop, explaining what it thinks it is doing
and with what input and with what result.  I just don't know how to
add that to scheme.

jim


On Tue, 2020-02-18 at 16:53 -0600, Tommy Trussell wrote:
> On Tue, Feb 18, 2020 at 4:07 PM crazylyle <lyle at austin.rr.com> wrote:
> > So you are suggesting reducing the size of our QIF files to a small
> > debuggable size.
> > 
> > My QIF file is 653,808 lines long.  About 2^20.  So just using a binary
> > search would
> > take at least 20 trials to find the first line that it fails on.  Not
> > exactly something that
> > seems practical.  And that's just for my input file.
> > 
> > And of course, just feeding a partial file might produce a failure, but a
> > failure of a 
> > different kind, so this won't necessarily get us anywhere.
> 
> What bisections or limited imports have you tried? 
> 
> If you haven't already, try going into Quicken and exporting one day, one
> month, one year, one account... a smaller test batch and see if you get the
> same error.
> 
> I know years ago (maybe 2012 or earlier) when I switched from Quicken to
> GnuCash I found that Quicken created many odd little problems that were just
> not worth the effort to clean up. SO I exported all my data (going back to
> about 1989) into one QIF file that I never bothered to import, but the data
> is still there if I need it. As you know, QIF is a text file, so you can
> search for transactions with a text editor.
> 
> Then I went back and exported just the previous one or two years from
> Quicken (so for you it might be 2019 and 2020-to-date) and imported only
> that data into GnuCash. In order to make the import work its best, I used a
> multi-step process, where I created the most-used accounts first, did a few
> test imports, fixed a few small things in Quicken, then did my import. I
> cleaned up the few remaining import problems in GnuCash and haven't looked
> back.
> 
> UNFORTUNATELY for you, most of us who make the conversion never go back and
> try the latest (there's no need) so the developers depend upon folks who
> encounter problems to fix them as they arise. 
> 
>  
> > Please remember to CC this list on all your replies.
> > You can do this by using Reply-To-List or Reply-All.