New web-banking -> OFX gateway (expensify.com); how to create importer?

Wed Dec 12 13:13:45 EST 2007

No offense to you, but I'm not sure I'd trust a third-party website
with my banking login information.

-derek

"David Barrett" <dbarrett at quinthar.com> writes:

> Thanks for the tips.  I've got the code and am attempting to build, but
> before I get too deep into it I'm curious if I build this, will anybody use
> it?  So a question to anyone who gets this:
>
> Are you interested in an importer that downloads your transaction data
> straight from your banking website?
>
> Basically, you'll be prompted for your bank's domain name, as well as the
> name/password you use to log on to your banking website, and it'll download
> all the accounts it can find and import them into GnuCash -- behaving just
> as if you had manually imported an OXF file containing all your accounts and
> transactions.
>
> To see what I mean, I've got a basic web-based interface up at
> expensify.com; the plugin would simply provide an interface to this via a
> new "File :: Import" command.
>
> If I build this, will you use it?
>
> Thanks!
>
> -david
>
> PS: I'm going to cross-post this on the user list to see if there's any
> interest there.
>
>> -----Original Message-----
>> From: Christian Stimming [mailto:stimming at tuhh.de]
>> Sent: Monday, December 10, 2007 8:00 AM
>> To: gnucash-devel at gnucash.org
>> Cc: David Barrett
>> Subject: Re: New web-banking -> OFX gateway (expensify.com); how to create
>> importer?
>> 
>> Hi David,
>> 
>> Am Montag, 10. Dezember 2007 10:44 schrieb David Barrett:
>> > As background, I didn't like how many banks either charge you extra to
>> > download OFX data, or simply don't offer it all.  So I built a gateway
>> to
>> > scrape banking websites for transactions and return them as OFX files.
>> 
>> That's an interesting project. I'm unsure about the security implications
>> of
>> what you offer here, and how you get those sorted out - but those
>> questions
>> are orthogonal to the implementation issues you asked for.
>> 
>> > As it stands, I'm at a loss to understand the current online banking
>> > functions and where to tie in.  I'm starting to dig through the code,
>> but
>> > it's rather daunting and I'd welcome any pointers you can offer.
>> 
>> Sure, *those* are the kind of questions we like to answer most. Don't
>> hesitate
>> to ask any further issues you might encounter.
>> 
>> > 1) Should I add this under "File :: Import" or "Tools :: Online Banking
>> > Setup"?  (I'm thinking "Files :: Import" because it not only looks
>> easier,
>> > but I'm really just building an "import" function -- it doesn't attempt
>> to
>> > simulate a true bidirectional OFX connection.)
>> 
>> File->Import is just fine. The "Online banking setup" is necessary because
>> (as
>> you might have guessed) for the two-way online banking some more setup
>> procedure is necessary. Did you have a look at
>> http://wiki.gnucash.org/wiki/AqBanking and/or the document
>> src/import-export/hbci/HACKING-HBCITEST.txt in the sources? You can use
>> those
>> to see what the so-called "HBCI" gnucash module will offer through the
>> aqbanking library, and what kind of setup it requires. (That module used
>> to
>> offer only the German HBCI online banking protocol, but as libaqbanking
>> grew
>> to support more than that, HBCI is merely the name of the module anymore
>> but
>> not its only supported protocol.)
>> 
>> File->Import is especially fine if the importer module doesn't need an
>> account
>> selected before it is called. I chose Actions->Online Banking as menu for
>> the
>> HBCI backend because those module functions expect an account to be
>> selected
>> before being called. This is achieved because those menu entries appear
>> only
>> in the register window, where the account of the currently opened register
>> is
>> taken as the selected one.
>> 
>> > 2) Is /src/import-export/ofx a good importer on which to base my code,
>> or
>> > is there some easier/cleaner place to start?  (Ideally, an importer that
>> > already pulls data down from an HTTP connection?)
>> 
>> Yes, maybe. Some of the code was written by its author at a time when he
>> still
>> got to learn modularized C coding style (that's why I say "maybe"), but
>> its
>> usage of the gnucash interface is pretty straightforward and is done
>> similarly by every other module as well.
>> 
>> > 3) Where is a good example showing some GnuCash networking code that I
>> can
>> > study?  (I'm very familiar with C/C++ network programming, but half the
>> > challenge is just getting the first line to compile; I'd like to see
>> some
>> > working code to get started.)
>> 
>> Surprise: GnuCash does *not* contain *any* networking code. All networking
>> connections that are being used for online banking are initiated and
>> managed
>> by libaqbanking (or its underlying libgwenhywfar), so gnucash doesn't have
>> to
>> do anything with networking at all.
>> 
>> If you want to integrate your http download into gnucash, your best bet is
>> probably to use some well-known HTTP client library and add this to the
>> gnucash dependency list. I'd recomment libcurl,
>> http://curl.haxx.se/libcurl/
>> and its documentation should get you started pretty fast.
>> http://curl.haxx.se/libcurl/c/libcurl-tutorial.html
>> 
>> > 4) Can you summarize for me at a high level the best way to get started
>> > writing a network-based OFX importer?  I really envision the following
>> > functions; any examples that already do each that I can study?
>> 
>> The approach sounds fine.
>> 
>> > 	- Add an entry to "File :: Import"
>> 
>> Each of the modules in gnucash is wrapped up in a datatype GncPlugin. For
>> your
>> use case, you should follow the coding in the src/import-export/csv
>> directory
>> because that one offers exactly this single menu entry as starting point.
>> (Note: This directory exists only in SVN trunk, not yet in the 2.2.x
>> releases.) Basically you'd copy over the files gnc-plugin-csv.* and
>> gnc-plugin-csv-ui.xml, where the latter file specifies the location of the
>> specific menu entry.
>> 
>> (It's up to your to either start yet another subdirectory of import-
>> export,
>> say, "http", or for faster startup simply add your menu entry into the ofx
>> or
>> csv subdirectory as well. I think it will be rather easy to move your code
>> into a separate subdirectory at a later point in time, if this turns out
>> to
>> be useful.)
>> 
>> > 	- Open a dialog asking for a Bank domain name, username, and
>> > password
>> 
>> Yes. Eventually, those values would be cached in the data file's "kvp
>> frame"
>> structures, similarly to those in the import-export/hbci/gnc-hbci-kvp.c
>> functions.
>> 
>> > 	- Open an HTTP connection to the data.expensify.com back end
>> 
>> Yes. I'd recommend you should use libcurl for this.
>> 
>> > 	- Hand the results to the OFX importer and let it take over
>> 
>> Yes. Instead of import-export/ofx which calls libofx_proc_file() you will
>> probably call libofx_proc_buffer(). That function will parse the ofx data
>> and
>> call specific callback functions for each ofx structure. Those callbacks
>> are
>> set in the ofx module as well. In your case, you will have to set the
>> libofx
>> callback functions similarly to the existing code in the import-export/ofx
>> module. For this reason, you can probably add your code into the ofx
>> module
>> directory and re-use most of the callback functions there.
>> 
>> > PS: Feel free to visit www.expensify.com to see it in action; use the
>> bank
>> > "demo.com" in order to get a feel for how it'd work, or type in your US
>> > Bank or Wells Fargo login credentials to see it live in action.  (Please
>> > excuse the certificate warnings; still getting that sorted out.)
>> >
>> > PPS: I'm reading through the documentation on the website, and the "QIF
>> > importer infrastructure" looks promising; is there anything else I
>> should
>> > be reading?
>> 
>> I'm afraid all of the documentation that refers to QIF is either very very
>> old
>> or very very unfinished. For QIF, we have one very old importer that works
>> most of the time (in "qif"), and another rewrite attempt that is very
>> unfinished (in "qif-import"). The rest of the importer documentation is in
>> the header files in import-export/*.h,
>> http://cvs.gnucash.org/docs/HEAD/group__Import__Export.html
>> Unfortunately the rest of the importing code is not very documented.
>> 
>> Regards,
>> 
>> Christian
>
> _______________________________________________
> gnucash-devel mailing list
> gnucash-devel at gnucash.org
> https://lists.gnucash.org/mailman/listinfo/gnucash-devel
>
>

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord at MIT.EDU                        PGP key available