ofx and generic import infrastructure

[Benoit Grégoire]

> Err... are you saying that the destination account matching *has* to be on
> the same page as on the transaction matching/duplicate detection? If that
> is what you want to say, then I definitely disagree... those are two
> different tasks and I can easily imagine those happening in different
> windows (or druid pages or whatever). Maybe you prefer to have both
> happening in one window (and I prefer the other) -- but if you think that
> there are important reasons for the one-window concept, can you elaborate
> more on that?

Ok, here is an example.  You regularly go to one of those  mega-stores where 
you can buy everything form Bread to computers, such as WalMart where you can 
buy grocery, audiovisual, computers, clothes, etc.  Monday you buy your 
groceries, Friday you got your paycheck and decide buy new TV.  The next week 
you download your credit card statement.  You now have two transactions with 
the EXACT same description, but only differing in the date and amount.  Since 
your groceries and TV are unlikely to go into the same expense account, you 
MUST have a way to override the destination account on a per transaction 
basis. 

Unless you mean for the first page to have one line per transaction, I don't 
see how you could handle it on your previous druid page.  The strings for 
which you would be picking an account would be identical (hence you could 
only select one account).  So, since you need to double check anyway and have 
an interface to override the selection, it seems kind of silly to have two 
separate, successive interfaces to do it.  Destination account matching 
(unlike source account matching) is dynamic in nature.  I changes over time, 
and there are a whole bunch of punctual exceptions.  

The WallMart example is not an isolated case, here are a few other stores 
where you would buy stuff that would not normally go in the same expense 
account.
-Drug stores (cosmetics, grocery and medication).  Here at least everyone are 
mandated by law to have medication insurance that will reimburse part of the 
price of prescription drugs.  Most people will handle this as a separate 
expense account.
-Hardware stores:  Furniture, construction materials and various household 
products.
-Electronics stores:  Furniture (Refrigerator, washing machines), Audiovisual 
and computers
-Basically any large surface store will have this problem to some extent.  And 
at least here in North America, that is where people buy most of their stuff.

> :-) Not really. My "Reconcile window invocation" happens in
>
> src/import-export/hbci/gnc-hbci-getbalance.c line 169. Not really much
> code, is it? :-) In the current HBCI implementation, the user simply gets a
> dialog window popping up in his face (line 148) asking "Reconcile now?
> Yes/No", and if the user answers yes,  the reconcile window is opened by
> recnWindowWithBalance from RecnWindow.h. That's all there is.

Darn, no code to steal :(  Unfortunately for me in OFX and QIF you can receive 
several balances in the same file.  Popping the same dialog four times in a 
row (once for each account) might trigger aggressive reactions by the user...  
So a smoother solution will have to be implemented for OFX and QIF.  But the 
smooth solution would require (ouch, it hurts to say this) a druid, to add a 
page after transaction matching.

> > Now, I've taken a lot of heat in the past for the event based, "pop in
> > your face" design of the current generic importer.  Derek and Christian
> > want it to be druid based.
>
> Ok, this point is still up for discussion, but there are several issues
> with it, and since you asked, I'm just going to state those:
>
> * One thing that keeps buggin me about the gnc_import_add_trans function is
> that it operates on static, hidden GUI data, which is obvious by the fact
> that it only takes one argument, the Transaction to add. IMHO the natural
> question here is "this thing adds transaction TO WHERE?" What I definitely
> would prefer here is a function like
>
> void gnc_import_add_trans(GenericImporter *importer, Transaction *trans);
>
> and having the additional function
>
> GenericImporter *gnc_import_importer_new (GtkWidget *parent);
>
> and so on. Note that this point is still totally independent from the
> Druid/One-Window discussion. It is, however, clearly not independent from
> the point you raised about different programming styles colliding here. :-)

I don't actually mind if it's done like this, but I think it's only moving the 
problem from one place to several place, which is usually a bad thing.  If 
the structure is created from outside the matcher, I will need to keep a 
static or global variable in the OFX module, so that each time a 
ofx_proc_trans_cb is called from LibOFX, the transactions will go to the same 
matcher.  Which implies that we will also have to define a 
gnc_import_importer_complete_cb(GenericImporter *importer) to tell the OFX 
module (and every other module now that I think of it) that the matcher has 
been closed by the user and should now be destroyed.  (Since there is is 
never any way the protocol specific module can know when the user has 
finished processing all transactions and closed the matcher).

One benefit however (as we discussed a few weeks ago) would be to allow OFX, 
QIF and HBCI imports to occur simultaneously, but each in their own window.  
IMHO it's not worth moving matcher creation and destruction outside the 
matcher and into each and every module, but I would not oppose a patch that 
does it (as long as it doesn't break OFX obviously).  

> * Same point (different programming styles): I would kindly ask you that if
> you define function prototypes in a header file, *please* put the
> implementation in a .c file *of the same name*. Otherwise the header file
> IMHO is almost useless -- I still have to grep for the actual
> implementation of that function, which is quite a bummer. Of course this
> can lead to the fact that you create a whole lot of header files, and maybe
> some of those are almost empty -- but IMHO this is totally valid and only
> reflects the *actual* *real* function interdependencies/ program structure.
> Note that this point can perfectly be done during the next days, since it
> shouldn't change the behaviour of the code at all.

I don't really understand the benefit.  To have a coherent and easy to 
understand API for the generic import infrastructure, I think all public 
functions should be in a single header file, so that this file can be 
included by the protocol specific module.  I've split the file to allow for 
easier reading and faster recompiles (it used to be a single file).  I think 
one header for each C file is usefull if you want to have prototypes for 
private functions (in which case you have one header per c file with the 
private function, and that header includes the header with the public 
functions), but I've never seen that anywhere in GnuCash.  In GnuCash all 
header files represent an API, and every c file have more functions that it's 
header file.  And I've never seen an API anywhere that forces the client to 
include a whole bunch of header files each containing one or two functions in 
order to use the API.  

I do however see your point about program structure, but I meant that to be 
reflected by the the c files.  I could write the filename where the function 
is implemented in the comments above the function.  I can't help but to think 
that If we split the headers in separate files and in the end we do a druid 
interface, the protocol module will have to include at least one header per 
page, which would be kind of funny :)

> confirming that). In other words: You shouldn't feel disappointed because
> nobody else is editing the files you do -- instead, this has just proven to
> be the most effective way of dividing up the work, especially in OpenSource
> projects. Not doing so (i.e. editing the *same* files) immediately requires
> a whole different level of
> understanding and communication, which IMHO might be necessary for a big,
> multi-purpose API (OpenHBCI) but not really for one part inside a big
> application. Gee. I'm pretty sure we'll come to a nicely working
> application in the end. Thanks for your comments.

Well, from that point of view, I think you are right.  But it would still have 
been nice to have negative comments on the API design while it was only that: 
a design document.  I might have ignored some of it, but i could have 
justified it as i went.  I received only thumbs up for the design document, 
and started receiving negative feedback only once I had working code, which 
was kind of demoralizing...

Oh well, the hard part is behind us in any case.  Once destination matching is 
implemented, I'll be able to turn my attention back to LibOFX, for which I 
have big projects (And since GnuCash is my testbed, it will also gain OFX and 
QIF export, and possibly OFC import in the process...)  Hopefully then we 
will soon have true interoperability between open source financial software!