New transaction matcher

[Christian Stimming]

Benoit Grégoire wrote:
 > Considering the above and the fact that we are way too close to a
 > release,  I think we no longer have a choice, we must fork.  I suggest
 > that you swallow a copy of gnc-gen-transaction.c back into the HBCI
 > module.

That's fine with me. What kind of matcher are you going to use? If you 
just return back to your original GUI, the fork would simply consist of 
reverting the 3 changed lines in gnc-ofx-import.c. I would prefer to 
have the gnc-gen-transaction.c still in the import-export directory. 
When splitting, you would simply not use that from the ofx directory. 
The HBCI directory then would continue to use it. The 
gnc-gen-transaction.c will also continue to use 90% of the code in 
Transaction-matcher.c.

My latest changes in Transaction-matcher.c (ChangeLog comment: "to ease 
a file split") do not necessarily mean that this file should be split 
any time soon (into one file with GUI functions and another with the 
non-GUI function). Rather, these changes are more of a trial on whether 
this would be easy or not. And IMHO for most of these functions it is 
probably not really easy, so this doesn't need to happen anytime soon. 
Again, I'm fine with this kind of split -- the gnc-gen-transaction would 
then just represent another front-end GUI to the Transaction-matcher.c code.

>>>-There must be visual feedback that the match was inconclusive.
>>>
>>I disagree.
> 
> Your are going against your own reasoning.  In your current matcher, if the 
> match was not originally "good enough", you will add the transaction by 
> default (with no clue that the transaction might have been  a duplicate).


Oh, you are pointing out that in gnc-gen-transaction.c:397 I use the 
clear_threshold twice, and that I might rather use the add_threshold? 
Sorry, then I misunderstood you here. Sure, the current choice is 
probably bad, and the add_threshold is the better choice. I didn't think 
about that too much, so yes, you're right -- that way, for inconclusive 
matches the user would be signalled that matches have been found, and 
he'll click on these (10%) to double-check for the correct matches.

> 1)  ------------
> You assume the users have different needs because they are using HBCI vs OFX, 
> and HBCI allows more assumptions.  This is simply not true.  If your users 
> have simpler needs, it is because of the way you claim they do their 
> transactions in real life in germany  (No commercial ATMs, very short credit 
> card processing delay).  
> 
> As far as the import process is concerned, HBCI vs OFX only have the following 
> technical differences:
> -1-  OFX can guarantee uniqueness of transactions, HBCI can't (currently).
> -2-  A new account will never come up during the import process for HBCI, 
> while that is currently always the way to add a new account for OFX.
> -3- Splits in HBCI are never balanced, while they sometimes are in OFX (most 
> investment transactions, and interaccount transfers (not implemented yet)).  
> But from glancing at the HBCI spec, it seems to support all information 
> needed for investment transactions, so this difference is temporary at best.


There's a principal problem in our discussion here, namely, nobody 
outside Germany has actually ever used HBCI (include you and all other 
Gnucash developers). Therefore I have the annoying problem that I have 
to argue about some things over and over again, even though for me (and 
*every* other German developer *and* user) they are perfectly obvious. 
So I'm sorry if haven't explained everything related to HBCI enough in 
detail, but please understand that this is also getting a bit tiresome 
for me. That's why I'm actually not too unhappy about splitting the 
front-end GUI while still sharing common back-end, non-GUI code in these 
various flavors of transaction importers.

Now for the technical differences: These three are correct. 
Additionally, there's one thing that I haven't mentioned previously 
(sorry for that):

* In HBCI, the date range of imported transactions is known in advance. 
I.e. the protocol-code says to the bank server: "give me all 
transactions starting from Nov 21 until now". And of course the protocol 
code saves this date-of-last-retrieval (in the account's kvp).

Therefore not only the 90%-new assumption will be fulfilled easily, but 
for the rest the user already knows the timeframe that's relevant in 
this importing process. I.e., I think that even before the import 
starts, the user already knows which transactions are going to be 
detected as duplicates. That's why I think the user doesn't need much 
more than to see the list of imported transactions, and then most of the 
time he knows already the transactions that should be identified as 
duplicates. Now I admit that's a rather weak argument as it doesn't 
scale at all for importing >20 transactions at once. And your point 
above about that I chose the wrong threshold is correct. Nevertheless I 
consider this as quite important when talking about the actual user 
experience for HBCI and importing.

I'd slightly rephrase you point 3: Any transaction import in HBCI always 
imports only one Split. Or even more general: any transaction import in 
HBCI always results in the list of Splits for the one known account. 
This is totally independent of the type of accounts -- I don't see the 
need for any special treatment of investment accounts here. So yes, this 
will continue to hold.

If, at a later point in time, the hbci-code is able to execute the 
import for several accounts in a row, then interaccount transfers might 
show up. On the other hand this can be dealt with inside the HBCI code: 
The source account is known, and it is known by its absolute real-life 
"address" (account #, bank code, country code). The destination account 
is known as well by its absolute real-life address. Therefore 
destination accounts that are HBCI-enabled and exist in the same book 
can be identified without any fuzzyness or uncertainty. Since this is 
yet another HBCI peculiarity, I would leave this step to the 
hbci-protocol code and *not* expect this from the generic transaction 
importer.


> I now compile with 
> HBCI support, and I'll try to maintain compatibility as much as possible.  


That's a good thing (tm) in either case :-)))) Thanks for installing 
openhbci. :-)


Christian