New web-banking -> OFX gateway (expensify.com); how to create importer?

[David Barrett]

Thanks for the tips.  I've got the code and am attempting to build, but
before I get too deep into it I'm curious if I build this, will anybody use
it?  So a question to anyone who gets this:

Are you interested in an importer that downloads your transaction data
straight from your banking website?

Basically, you'll be prompted for your bank's domain name, as well as the
name/password you use to log on to your banking website, and it'll download
all the accounts it can find and import them into GnuCash -- behaving just
as if you had manually imported an OXF file containing all your accounts and
transactions.

To see what I mean, I've got a basic web-based interface up at
expensify.com; the plugin would simply provide an interface to this via a
new "File :: Import" command.

If I build this, will you use it?

Thanks!

-david

PS: I'm going to cross-post this on the user list to see if there's any
interest there.

> -----Original Message-----
> From: Christian Stimming [mailto:stimming at tuhh.de]
> Sent: Monday, December 10, 2007 8:00 AM
> To: gnucash-devel at gnucash.org
> Cc: David Barrett
> Subject: Re: New web-banking -> OFX gateway (expensify.com); how to create
> importer?
> 
> Hi David,
> 
> Am Montag, 10. Dezember 2007 10:44 schrieb David Barrett:
> > As background, I didn't like how many banks either charge you extra to
> > download OFX data, or simply don't offer it all.  So I built a gateway
> to
> > scrape banking websites for transactions and return them as OFX files.
> 
> That's an interesting project. I'm unsure about the security implications
> of
> what you offer here, and how you get those sorted out - but those
> questions
> are orthogonal to the implementation issues you asked for.
> 
> > As it stands, I'm at a loss to understand the current online banking
> > functions and where to tie in.  I'm starting to dig through the code,
> but
> > it's rather daunting and I'd welcome any pointers you can offer.
> 
> Sure, *those* are the kind of questions we like to answer most. Don't
> hesitate
> to ask any further issues you might encounter.
> 
> > 1) Should I add this under "File :: Import" or "Tools :: Online Banking
> > Setup"?  (I'm thinking "Files :: Import" because it not only looks
> easier,
> > but I'm really just building an "import" function -- it doesn't attempt
> to
> > simulate a true bidirectional OFX connection.)
> 
> File->Import is just fine. The "Online banking setup" is necessary because
> (as
> you might have guessed) for the two-way online banking some more setup
> procedure is necessary. Did you have a look at
> http://wiki.gnucash.org/wiki/AqBanking and/or the document
> src/import-export/hbci/HACKING-HBCITEST.txt in the sources? You can use
> those
> to see what the so-called "HBCI" gnucash module will offer through the
> aqbanking library, and what kind of setup it requires. (That module used
> to
> offer only the German HBCI online banking protocol, but as libaqbanking
> grew
> to support more than that, HBCI is merely the name of the module anymore
> but
> not its only supported protocol.)
> 
> File->Import is especially fine if the importer module doesn't need an
> account
> selected before it is called. I chose Actions->Online Banking as menu for
> the
> HBCI backend because those module functions expect an account to be
> selected
> before being called. This is achieved because those menu entries appear
> only
> in the register window, where the account of the currently opened register
> is
> taken as the selected one.
> 
> > 2) Is /src/import-export/ofx a good importer on which to base my code,
> or
> > is there some easier/cleaner place to start?  (Ideally, an importer that
> > already pulls data down from an HTTP connection?)
> 
> Yes, maybe. Some of the code was written by its author at a time when he
> still
> got to learn modularized C coding style (that's why I say "maybe"), but
> its
> usage of the gnucash interface is pretty straightforward and is done
> similarly by every other module as well.
> 
> > 3) Where is a good example showing some GnuCash networking code that I
> can
> > study?  (I'm very familiar with C/C++ network programming, but half the
> > challenge is just getting the first line to compile; I'd like to see
> some
> > working code to get started.)
> 
> Surprise: GnuCash does *not* contain *any* networking code. All networking
> connections that are being used for online banking are initiated and
> managed
> by libaqbanking (or its underlying libgwenhywfar), so gnucash doesn't have
> to
> do anything with networking at all.
> 
> If you want to integrate your http download into gnucash, your best bet is
> probably to use some well-known HTTP client library and add this to the
> gnucash dependency list. I'd recomment libcurl,
> http://curl.haxx.se/libcurl/
> and its documentation should get you started pretty fast.
> http://curl.haxx.se/libcurl/c/libcurl-tutorial.html
> 
> > 4) Can you summarize for me at a high level the best way to get started
> > writing a network-based OFX importer?  I really envision the following
> > functions; any examples that already do each that I can study?
> 
> The approach sounds fine.
> 
> > 	- Add an entry to "File :: Import"
> 
> Each of the modules in gnucash is wrapped up in a datatype GncPlugin. For
> your
> use case, you should follow the coding in the src/import-export/csv
> directory
> because that one offers exactly this single menu entry as starting point.
> (Note: This directory exists only in SVN trunk, not yet in the 2.2.x
> releases.) Basically you'd copy over the files gnc-plugin-csv.* and
> gnc-plugin-csv-ui.xml, where the latter file specifies the location of the
> specific menu entry.
> 
> (It's up to your to either start yet another subdirectory of import-
> export,
> say, "http", or for faster startup simply add your menu entry into the ofx
> or
> csv subdirectory as well. I think it will be rather easy to move your code
> into a separate subdirectory at a later point in time, if this turns out
> to
> be useful.)
> 
> > 	- Open a dialog asking for a Bank domain name, username, and
> > password
> 
> Yes. Eventually, those values would be cached in the data file's "kvp
> frame"
> structures, similarly to those in the import-export/hbci/gnc-hbci-kvp.c
> functions.
> 
> > 	- Open an HTTP connection to the data.expensify.com back end
> 
> Yes. I'd recommend you should use libcurl for this.
> 
> > 	- Hand the results to the OFX importer and let it take over
> 
> Yes. Instead of import-export/ofx which calls libofx_proc_file() you will
> probably call libofx_proc_buffer(). That function will parse the ofx data
> and
> call specific callback functions for each ofx structure. Those callbacks
> are
> set in the ofx module as well. In your case, you will have to set the
> libofx
> callback functions similarly to the existing code in the import-export/ofx
> module. For this reason, you can probably add your code into the ofx
> module
> directory and re-use most of the callback functions there.
> 
> > PS: Feel free to visit www.expensify.com to see it in action; use the
> bank
> > "demo.com" in order to get a feel for how it'd work, or type in your US
> > Bank or Wells Fargo login credentials to see it live in action.  (Please
> > excuse the certificate warnings; still getting that sorted out.)
> >
> > PPS: I'm reading through the documentation on the website, and the "QIF
> > importer infrastructure" looks promising; is there anything else I
> should
> > be reading?
> 
> I'm afraid all of the documentation that refers to QIF is either very very
> old
> or very very unfinished. For QIF, we have one very old importer that works
> most of the time (in "qif"), and another rewrite attempt that is very
> unfinished (in "qif-import"). The rest of the importer documentation is in
> the header files in import-export/*.h,
> http://cvs.gnucash.org/docs/HEAD/group__Import__Export.html
> Unfortunately the rest of the importing code is not very documented.
> 
> Regards,
> 
> Christian