Strange behavior importing QFX files...

[David Reiser]

On Sep 24, 2008, at 1:20 AM, Trey Blancher wrote:

> I'm using GnuCash 2.2.4, and the initial QFX import into a brand new
> GnuCash file works.  Everything in my checking account reconciles
> nicely.  The following month, when I go to import this months QFX file
> from my bank, GnuCash "misses", for lack of a better term, some key
> transactions.  I've examined the raw QFX files, and the transactions
> are indeed present, but GnuCash simply does not add them to the
> register.  The missed transactions invariably are one of two classes
> of transaction:  they're a repeating transaction with the same dollar
> amount each month (e.g. my rent, and my student loan are the same
> value each month), or they're one of my several small (i.e.
> $1-$5/transaction) transfers into my savings account (my bank has a
> program where every purchase I make with my check card causes $1 to be
> transferred to a high-interest savings account).
>
> I've had this problem for several months if not more than a year
> (actually, ever since I started importing from my bank sometime last
> year).  I don't know how many versions of GnuCash I've tried, but it
> has happened with every one.  I've examined the transaction IDs
> (<FITID>) for the previous and current months for the missed
> transactions, and as far as I can tell they're different.  Doing a
> spot check on my monthly rent, and they only differ by one digit, but
> they are different (2008070101 for one, and 2008080101 for the
> second), so I don't see why GnuCash should skip them.  I'm willing to
> supply my QFX files demonstrating the behavior, if someone can point
> me to a good QFX file format reference so I can properly sanitize
> them.  I think I also tried OFX files, and got the same behavior.
>
> Any help would be appreciated.
>
> Thanks!
>
> --  
> Trey Blancher

The matcher has problems with competing matches, especially (only?)  
with cases where some transactions already exist in the register and  
others are only in the import list.

If you have a transaction in your register

9/15/2008 Joe's Bar and Grill $37.95

and the ofx file has both

9/16/2008 JOES D34567 $37.95
9/17/2008 KMart #4567 $37.50

the initial matcher window will show that both the import stream  
transactions have a potential match. Possibly one or both of them will  
be highlighted in red because the system doesn't see a close enough  
match because the description doesn't match in either case, and if the  
date moves a day because of a credit card posting delay, then the date  
won't match exactly either. The amounts match within the ATM fee  
threshold in both cases. And the dates do match within the hard coded  
'float' window (2 weeks, I think), so they'll get partial credit for  
that.

If you check the R column for both (a good thing to do if both  
transactions are already in the register, despite a marginal match  
score), then when the actual import occurs, only one of the  
transactions will end up in your file because the first ofx  
transaction matches the register transaction (even if it is the wrong  
amount transaction. Furthermore, the importer keeps your entered data  
instead of 'correcting' it with the import values.) When the second  
match gets parsed, there's no matching transaction left, so the import  
throws it away. (I suppose it could be serially matching 2 ofx  
transactions to one register transaction, since the only visible  
effect is the same -- the transaction is marked "c". I'd have to look  
at the raw data file to see which OFX transaction ID ends up  
associated with the register transaction.)

Something that just occurred to me today is to try this after setting  
the ATM threshold in preferences to 0.00. That might force the ofx  
transaction that's missing in the register to appear as A instead of R  
in the initial match window. That at least would improve some of the  
imports.

I'm not at all sure that this description covers all of my import  
problems, but it is one that I can reproduce reliably.

Dave
--
David Reiser
dbreiser at earthlink.net