New web-banking -> OFX gateway (expensify.com); how to create importer?

[Christian Stimming]

Hi David,

Am Montag, 10. Dezember 2007 10:44 schrieb David Barrett:
> As background, I didn't like how many banks either charge you extra to
> download OFX data, or simply don't offer it all.  So I built a gateway to
> scrape banking websites for transactions and return them as OFX files.

That's an interesting project. I'm unsure about the security implications of 
what you offer here, and how you get those sorted out - but those questions 
are orthogonal to the implementation issues you asked for.

> As it stands, I'm at a loss to understand the current online banking
> functions and where to tie in.  I'm starting to dig through the code, but
> it's rather daunting and I'd welcome any pointers you can offer.

Sure, *those* are the kind of questions we like to answer most. Don't hesitate 
to ask any further issues you might encounter.

> 1) Should I add this under "File :: Import" or "Tools :: Online Banking
> Setup"?  (I'm thinking "Files :: Import" because it not only looks easier,
> but I'm really just building an "import" function -- it doesn't attempt to
> simulate a true bidirectional OFX connection.)

File->Import is just fine. The "Online banking setup" is necessary because (as 
you might have guessed) for the two-way online banking some more setup 
procedure is necessary. Did you have a look at 
http://wiki.gnucash.org/wiki/AqBanking and/or the document 
src/import-export/hbci/HACKING-HBCITEST.txt in the sources? You can use those 
to see what the so-called "HBCI" gnucash module will offer through the 
aqbanking library, and what kind of setup it requires. (That module used to 
offer only the German HBCI online banking protocol, but as libaqbanking grew 
to support more than that, HBCI is merely the name of the module anymore but 
not its only supported protocol.)

File->Import is especially fine if the importer module doesn't need an account 
selected before it is called. I chose Actions->Online Banking as menu for the 
HBCI backend because those module functions expect an account to be selected 
before being called. This is achieved because those menu entries appear only 
in the register window, where the account of the currently opened register is 
taken as the selected one.

> 2) Is /src/import-export/ofx a good importer on which to base my code, or
> is there some easier/cleaner place to start?  (Ideally, an importer that
> already pulls data down from an HTTP connection?)

Yes, maybe. Some of the code was written by its author at a time when he still 
got to learn modularized C coding style (that's why I say "maybe"), but its 
usage of the gnucash interface is pretty straightforward and is done 
similarly by every other module as well.

> 3) Where is a good example showing some GnuCash networking code that I can
> study?  (I'm very familiar with C/C++ network programming, but half the
> challenge is just getting the first line to compile; I'd like to see some
> working code to get started.)

Surprise: GnuCash does *not* contain *any* networking code. All networking 
connections that are being used for online banking are initiated and managed 
by libaqbanking (or its underlying libgwenhywfar), so gnucash doesn't have to 
do anything with networking at all. 

If you want to integrate your http download into gnucash, your best bet is 
probably to use some well-known HTTP client library and add this to the 
gnucash dependency list. I'd recomment libcurl,  http://curl.haxx.se/libcurl/ 
and its documentation should get you started pretty fast. 
http://curl.haxx.se/libcurl/c/libcurl-tutorial.html

> 4) Can you summarize for me at a high level the best way to get started
> writing a network-based OFX importer?  I really envision the following
> functions; any examples that already do each that I can study?

The approach sounds fine.

> 	- Add an entry to "File :: Import"

Each of the modules in gnucash is wrapped up in a datatype GncPlugin. For your 
use case, you should follow the coding in the src/import-export/csv directory 
because that one offers exactly this single menu entry as starting point. 
(Note: This directory exists only in SVN trunk, not yet in the 2.2.x 
releases.) Basically you'd copy over the files gnc-plugin-csv.* and 
gnc-plugin-csv-ui.xml, where the latter file specifies the location of the 
specific menu entry.

(It's up to your to either start yet another subdirectory of import-export, 
say, "http", or for faster startup simply add your menu entry into the ofx or 
csv subdirectory as well. I think it will be rather easy to move your code 
into a separate subdirectory at a later point in time, if this turns out to 
be useful.)

> 	- Open a dialog asking for a Bank domain name, username, and
> password

Yes. Eventually, those values would be cached in the data file's "kvp frame" 
structures, similarly to those in the import-export/hbci/gnc-hbci-kvp.c 
functions.

> 	- Open an HTTP connection to the data.expensify.com back end

Yes. I'd recommend you should use libcurl for this.

> 	- Hand the results to the OFX importer and let it take over

Yes. Instead of import-export/ofx which calls libofx_proc_file() you will 
probably call libofx_proc_buffer(). That function will parse the ofx data and 
call specific callback functions for each ofx structure. Those callbacks are 
set in the ofx module as well. In your case, you will have to set the libofx 
callback functions similarly to the existing code in the import-export/ofx 
module. For this reason, you can probably add your code into the ofx module 
directory and re-use most of the callback functions there.

> PS: Feel free to visit www.expensify.com to see it in action; use the bank
> "demo.com" in order to get a feel for how it'd work, or type in your US
> Bank or Wells Fargo login credentials to see it live in action.  (Please
> excuse the certificate warnings; still getting that sorted out.)
>
> PPS: I'm reading through the documentation on the website, and the "QIF
> importer infrastructure" looks promising; is there anything else I should
> be reading?

I'm afraid all of the documentation that refers to QIF is either very very old 
or very very unfinished. For QIF, we have one very old importer that works 
most of the time (in "qif"), and another rewrite attempt that is very 
unfinished (in "qif-import"). The rest of the importer documentation is in 
the header files in import-export/*.h, 
http://cvs.gnucash.org/docs/HEAD/group__Import__Export.html
Unfortunately the rest of the importing code is not very documented.

Regards,

Christian