Generic transaction import and qif-import

[Christian Stimming]

Here's an edited log of our IRC conversation about how to reuse the 
transaction import code from the QIF-Importer for other import methods, 
notably HBCI.

The question was raised whether gnome-guile is included in RedHat 7.3, 
does anybody know about that?

Log:

** qif-import module and qif-io-core module; duplicate detection code

<cstim> bgrib: did you start the qif-io-core module? I wonder why the 
development on it stopped some time ago.
<bgrib> yes, I started the qif-io-core.  It was intended to be a 
testable/flexible replacement for the current importer, and to support 
output; also, to detach the GUI from the import/export logic.
<bgrib> um, I'm not sure how to put this exactly :) but you may have 
noticed that *all* development by me, dave, rlb, and dres has pretty 
much stopped on gnucash-the-personal-finance-program.  just no time to 
work on it for the foreseeable future.
<cstim> For hbci transaction import, I would like to borrow the code for 
transaction duplicate detection.
<bgrib> it's completely separate from the QIF part of the code, so it 
should be easy to reuse.  look at qif-merge-groups.scm.  it detects 
duplicates between 2 gnucash account groups.
<cstim> I think I would like to refactor the "transaction import" code 
(including duplicate detection) into a new, import-format-independent 
module, say, "tx-import", and then have both QIF and HBCI (and possibly 
QFX) use it.
<bgrib> I somewhat agree.. there still needs to be a lot of 
format-dependent logic.  but it could be pushed down to a lower level, 
and leave a higher level for generic import. but then the "intermediate 
representation" would be a gnucash account group, and nothing's left to 
do except remove duplicates.
<bgrib> the qif-io-core code is probably the place to start if you are 
doing a refactoring.  it's a much cleaner base for working with the QIF 
format. It can almost do the whole import process as  is (or could the 
last time I looked at it) and there are lots of test cases showing how 
it works.
<cstim> oh. do you recall what parts of qif-io-core *didn't* work?
<bgrib> it's just not finished ... don't remember exactly how far along 
it is.
<cstim> So that sounds like for refactoring of existing code, I should 
rather use the code from qif-io-core instead of qif-import?
<bgrib> qif-io-core fits better with your new model.  the parsing is 
done in a way that makes more sense and is testable.  the whole model 
for duplicate detection is more sensible (converts to gnc transactions 
as soon as possible, and does all duplicate logic on gnc transactions... 
old code does some duplicate removal before converting, some after )
<bgrib> so, yes... use qif-io-code if possible.  but you will have to do 
a little more work to merge parts from qif-import that aren't 
implemented in qif-io-core yet, and you will have to make a GUI.

** Making a GUI

<cstim> what is our preferred way of making a GUI? Following 
qif-import/druid-qif-import.c and dialog-account-picker.c as an example 
or rather something different?
<bgrib> cstim: in my opinion, there is no reason to write any new code 
for gnucash in C, unless you have to make an engine extension.  the GUI 
should be done completely in Glade, and all the GUI logic using the 
guile-gnome wrappers.
<bgrib> of course, the details are a little more complicated :)
<bgrib> the guile-gnome wrappers are not 100% complete, and sometimes I 
find that I have to write a little C stub function library and g-wrap 
spec file to get access to some gnome/gtk things
<cstim> bgrib: are there examples in gnucash that I could follow?
<warlord> cstim: nope, currently gnucash does not use gnome-guile
<bgrib> maybe it's not worth another dependency.
<bgrib> in my opinion, it is.  it's a far,far better way to write gui code.
<warlord> bgrib: does Red Hat 7.3 distribute gnome-guile?
<warlord> (if not, then we're not going to use it for gnucash 1.8)
<bgrib> I don't pay much attention to red hat, sorry.
<cstim> errr... so if I don't have gnome-guile, how do I make a GUI then?
<bgrib> cstim: you can do it the way the current qif importer does, with 
glade and C
<warlord> cstim: glade, C, and g-wrap

** Duplicate detection code, again

<warlord> However, I do believe that the various import routines _may_ 
want to do some duplicate detection early and some later.
<warlord> regarding duplicate detection: OFX has uniqueids in each 
transaction which can help with duplicate detection early-on.  Is it 
worth doing all the processing to convert to a gnc-txn and then 
detecting it rather than doing the 'easy' detection early in the 
process?  If we do it later, then the 'generic' code is going to need to 
know about HBCI txn-ids, OFX txn-ids, etc.
<warlord> methinks that _most_ of the process can be generic, and there 
are only a few specific druid pages.
<warlord> (and even some of those specific pages might be able to be 
reused, like "username/password" ;)
<cstim> warlord: yes, I think that the GUI of transactions import should 
be as generic as possible.  and I can imagine a very generic duplicate 
detection as well.
<cstim> There will always be manually entered transactions that are 
duplicates of automatically imported ones (think of money withdrawals 
that I'll manually book in my Assets:Cash), so a generic GUI will always 
be needed.
<cstim> That's why I would go for a generic transaction importer in a 
new module tx-import, which should be including duplicate detection.
<warlord> cstim: oh, of course...  All I was saying is that I think 
duplicate detection should be done in BOTH places..
<warlord> It obviously needs to happen at the generic level.
<warlord> But it should also be able to happen in the specific level, too.
<cstim> warlord: ah, ok.