[GNC-dev] Hello and ofx version 2

[Jim DeLaHunt]

Hello, Steve, and welcome to the Gnucash-Devel list!

On 2020-05-31 10:00, Steve Burns wrote:
> Hello everyone, my name is Steve.  This is my first post here.
> I have been using GnuCash for many years.  It reminds me of Quicken from
> the 90's, only it's much better.
>
> I have recently come across an issue with importing QFX / OFX files in the
> U.S.
> Importing "version 1" style .qfx (like chase credit cards or my credit
> union) work fine.  The qfx file starts off with:
> OFXHEADER:100
> DATA:OFXSGML
> VERSION:102
> ...
>
> As of 2020, American Express now creates qfx files with a new format.
> They allowed the old format output for a few months, but now it is
> completely gone.
> The new format starts off with:
> <?xml version="1.0" standalone="no"?><?OFX OFXHEADER="200" VERSION="202" ...


This looks to me like American Express used to generate QFX/OFX version 
1 files, and now generate QFX/OFX version 2 files.

> GnuCash refuses to extract any data from this file.


That's a problem!  If you rely on OFX file import, as I do, I can 
understand that this obstacle is very frustrating.


> I haven't looked through the GnuCash source, but my guess is that we 
> need to:
> 1) Ignore carriage returns, or make them optional.  Use spaces and tags to
> delimit.  If the parsing code is line-based, this may be some work.
> 2) Be able to handle version 2 headers.  They contain no useful number
> data, so this may be straightforward.
> 3) Ignore more ending tags, if necessary.  Maybe nothing to do here.
>
> I think this is a necessary or high-need improvement to GnuCash.

I'm not the resident OFX expert, but my understanding is that GnuCash 
should already have the capability to import both version 1 and version 
2 OFX files. I looked for a feature list statement like "GnuCash can 
import QFX" and "GnuCash can import OFX version 2.2", and din't find it. 
But I see a bug report against OFX 2.2 file import 
<https://bugs.gnucash.org/show_bug.cgi?id=796782>, which implies that it 
generally works, though with some exceptions. LibOFX 
<https://github.com/libofx/libofx/>, which I understand is the library 
which GnuCash calls to do the actual import, has mentions of OFX 2 
support from almost its first version.

And Quicken needs to straight up support the file format, lack of 
carriage returns and version headers and end tags and all. There is a 
lot to it, the OFX version 2.2 spec 
<https://www.ofx.net/downloads/OFX%202.2.pdf> is 667 pages long. QFX is 
not the same as OFX. I have heard people refer to QFX as Quicken's 
bowlderised version of OFX. But for most purposes, I understand they are 
pretty interchangeable.

I understand why you would take the expedient of slicing up an OFX 
version 2 file to make it look more like an OFX version 1 file. I have 
to do an expedient slicing-up of OFX files from one of my banks to work 
around their mistakes in the files they generate, so I sympathise. 
However, this expedient approach is not the right way to get something 
as important as OFX support into GnuCash.

I suggest you proceed by treating this as a user-support situation 
instead of a feature-development situation. Ask on the gnucash-user list 
if anyone is also using QFX from American Express, and if it works for 
them. See if any of your other financial providers deliver OFX 2 files, 
and see if GnuCash imports those. File a bug report 
<https://bugs.gnucash.org/page.cgi?id=bug-writing.html> in the Gnucash 
issue tracker. And especially, see if you can modify one of the failing 
files so that you can post it to the bug report as a way for others to 
reproduce the problem. This will require you to strip out the sensitive 
information, while preserving the ability to demonstrate the failure.

I hope this is helpful. I'm looking forward to your contributions to 
GnuCash!

Best regards,
       —Jim DeLaHunt, software engineer, Vancouver, Canada