New web-banking -> OFX gateway (expensify.com); how to create importer?
stimming at tuhh.de
Mon Dec 10 11:00:24 EST 2007
Am Montag, 10. Dezember 2007 10:44 schrieb David Barrett:
> As background, I didn't like how many banks either charge you extra to
> download OFX data, or simply don't offer it all. So I built a gateway to
> scrape banking websites for transactions and return them as OFX files.
That's an interesting project. I'm unsure about the security implications of
what you offer here, and how you get those sorted out - but those questions
are orthogonal to the implementation issues you asked for.
> As it stands, I'm at a loss to understand the current online banking
> functions and where to tie in. I'm starting to dig through the code, but
> it's rather daunting and I'd welcome any pointers you can offer.
Sure, *those* are the kind of questions we like to answer most. Don't hesitate
to ask any further issues you might encounter.
> 1) Should I add this under "File :: Import" or "Tools :: Online Banking
> Setup"? (I'm thinking "Files :: Import" because it not only looks easier,
> but I'm really just building an "import" function -- it doesn't attempt to
> simulate a true bidirectional OFX connection.)
File->Import is just fine. The "Online banking setup" is necessary because (as
you might have guessed) for the two-way online banking some more setup
procedure is necessary. Did you have a look at
http://wiki.gnucash.org/wiki/AqBanking and/or the document
src/import-export/hbci/HACKING-HBCITEST.txt in the sources? You can use those
to see what the so-called "HBCI" gnucash module will offer through the
aqbanking library, and what kind of setup it requires. (That module used to
offer only the German HBCI online banking protocol, but as libaqbanking grew
to support more than that, HBCI is merely the name of the module anymore but
not its only supported protocol.)
File->Import is especially fine if the importer module doesn't need an account
selected before it is called. I chose Actions->Online Banking as menu for the
HBCI backend because those module functions expect an account to be selected
before being called. This is achieved because those menu entries appear only
in the register window, where the account of the currently opened register is
taken as the selected one.
> 2) Is /src/import-export/ofx a good importer on which to base my code, or
> is there some easier/cleaner place to start? (Ideally, an importer that
> already pulls data down from an HTTP connection?)
Yes, maybe. Some of the code was written by its author at a time when he still
got to learn modularized C coding style (that's why I say "maybe"), but its
usage of the gnucash interface is pretty straightforward and is done
similarly by every other module as well.
> 3) Where is a good example showing some GnuCash networking code that I can
> study? (I'm very familiar with C/C++ network programming, but half the
> challenge is just getting the first line to compile; I'd like to see some
> working code to get started.)
Surprise: GnuCash does *not* contain *any* networking code. All networking
connections that are being used for online banking are initiated and managed
by libaqbanking (or its underlying libgwenhywfar), so gnucash doesn't have to
do anything with networking at all.
If you want to integrate your http download into gnucash, your best bet is
probably to use some well-known HTTP client library and add this to the
gnucash dependency list. I'd recomment libcurl, http://curl.haxx.se/libcurl/
and its documentation should get you started pretty fast.
> 4) Can you summarize for me at a high level the best way to get started
> writing a network-based OFX importer? I really envision the following
> functions; any examples that already do each that I can study?
The approach sounds fine.
> - Add an entry to "File :: Import"
Each of the modules in gnucash is wrapped up in a datatype GncPlugin. For your
use case, you should follow the coding in the src/import-export/csv directory
because that one offers exactly this single menu entry as starting point.
(Note: This directory exists only in SVN trunk, not yet in the 2.2.x
releases.) Basically you'd copy over the files gnc-plugin-csv.* and
gnc-plugin-csv-ui.xml, where the latter file specifies the location of the
specific menu entry.
(It's up to your to either start yet another subdirectory of import-export,
say, "http", or for faster startup simply add your menu entry into the ofx or
csv subdirectory as well. I think it will be rather easy to move your code
into a separate subdirectory at a later point in time, if this turns out to
> - Open a dialog asking for a Bank domain name, username, and
Yes. Eventually, those values would be cached in the data file's "kvp frame"
structures, similarly to those in the import-export/hbci/gnc-hbci-kvp.c
> - Open an HTTP connection to the data.expensify.com back end
Yes. I'd recommend you should use libcurl for this.
> - Hand the results to the OFX importer and let it take over
Yes. Instead of import-export/ofx which calls libofx_proc_file() you will
probably call libofx_proc_buffer(). That function will parse the ofx data and
call specific callback functions for each ofx structure. Those callbacks are
set in the ofx module as well. In your case, you will have to set the libofx
callback functions similarly to the existing code in the import-export/ofx
module. For this reason, you can probably add your code into the ofx module
directory and re-use most of the callback functions there.
> PS: Feel free to visit www.expensify.com to see it in action; use the bank
> "demo.com" in order to get a feel for how it'd work, or type in your US
> Bank or Wells Fargo login credentials to see it live in action. (Please
> excuse the certificate warnings; still getting that sorted out.)
> PPS: I'm reading through the documentation on the website, and the "QIF
> importer infrastructure" looks promising; is there anything else I should
> be reading?
I'm afraid all of the documentation that refers to QIF is either very very old
or very very unfinished. For QIF, we have one very old importer that works
most of the time (in "qif"), and another rewrite attempt that is very
unfinished (in "qif-import"). The rest of the importer documentation is in
the header files in import-export/*.h,
Unfortunately the rest of the importing code is not very documented.
More information about the gnucash-devel