New transaction matcher

[Benoit Grégoire]

> > -There must be visual feedback that the match was inconclusive.
>
> I disagree. Right now, there are two possibilities: Either a good enough
> match was found -- then it's displayed there. Or no good enough match was
> found -- then the "New" column is marked. If the user thinks that there
> should have been a matching transaction, he should click on the "New"
> column and happily find his (fuzzy-matching) transaction in the
> matcher-dialog. IMHO this is just enough. Remember, in "The Usual Scenario"
> only 10% of the transactions are not new. Therefore I'm pretty sure that
> the duplicate-matching will, in some way or the other, be able to present
> enough potential duplicates to the user. Also remember that in HBCI, the
> amount of duplicate transactions match exactly in 99% of the cases.
> Therefore in HBCI, I really don't need much fuzzy matching here -- the
> exact matching account is already a good enough criterion.

Your are going against your own reasoning.  In your current matcher, if the 
match was not originally "good enough", you will add the transaction by 
default (with no clue that the transaction might have been  a duplicate).  So 
if your 90% new/10% duplicate ratio is typical (and I believe it is), it 
means that your users are going to to have to double check 90% of the 
transactions to see if they were indeed not a duplicate, since they have no 
info on which ones are a "close call".  So if you want to KISS and fast for 
90% of the time, using your current interface you should default to 
reconcile, so your users only have to double check ~10% of the transactions.

I still think it would be best to show the score of the best match, so the 
users only have to check the ~5% transactions for which it's hard to tell.

> > -The actions must go back to the main window.
>
> No :-)
>
> Seriously. I have written the message with "The Usual HBCI Scenario" to
> point out that I have a bunch of assumptions that I consider to hold when
> users are importing through HBCI. From that scenario, I derived the use
> cases which really boil down to the fact that the [HBCI] user does not need
> any further choices than the two shown in the main importer window.
> Therefore, if you propose GUI changes in that respect, then I can only
> agree if the HBCI code can specify configuration arguments so that the
> [HBCI] user won't get more choices than he does now.

Well, I'm sorry but with that attitude we can't get anywhere.   It seems to me 
now as tough you were never really ready to have a generic matcher.  Your 
point of view seems to be that if it's of no use for german users 90%
of the time, you don't want to see it, no matter how many hoops the user has 
to jump thru for the other 10%.  I'm sorry, but we can't build something 
GENERIC in these conditions, it a contradiction.  By definition there are 
compromises to be made and a cost in both coding complexity and interface 
complexity for building something truly generic.  We wanted to go this way 
(considering that interface complexity can be minimized by configuration) 
because we hope that having a single matcher for OFX HBCI and QIF will lead 
to more stability and better features on the long run.

Now there are two assumptions you seem to make that really annoy me and are 
poisoning this discussion (causing us to argue about things that are not 
really relevant)

1)  ------------
You assume the users have different needs because they are using HBCI vs OFX, 
and HBCI allows more assumptions.  This is simply not true.  If your users 
have simpler needs, it is because of the way you claim they do their 
transactions in real life in germany  (No commercial ATMs, very short credit 
card processing delay).  Even then, I sometimes think you are 
over-generalizing your scenario, but as long as the user can customize 
behavior, it's not too important.  

As far as the import process is concerned, HBCI vs OFX only have the following 
technical differences:
-1-  OFX can guarantee uniqueness of transactions, HBCI can't (currently).
-2-  A new account will never come up during the import process for HBCI, 
while that is currently always the way to add a new account for OFX.
-3- Splits in HBCI are never balanced, while they sometimes are in OFX (most 
investment transactions, and interaccount transfers (not implemented yet)).  
But from glancing at the HBCI spec, it seems to support all information 
needed for investment transactions, so this difference is temporary at best.

1 and 3 have some relevance to the transaction matcher, but haven't been the 
source of much of our disagreements, have they?  And since 3 can't be relied 
upon to hold in the future, only 1 is left, and it's satisfactory for now.

2)  ------------
Right now, you let the computer make a yes/no decision on wether the 
transaction has to be added as new no mater how unsure it is.

You claim that the user will just double check that the transactions are 
indeed new, but you've removed from the main window ALL the information that 
would allow the user to make that choice once the computer has decided that 
it should add as new.  I think you don't really expect the users to double 
check this into the transaction matcher on a regular basis, otherwise you 
wouldn't be arguing on hiding the match confidence or the "INCONCLUSIVE" 
action.

If the computer doesn't know, it MUST tell the user.

> Again I disagree. I want this GUI to be f**ing simple and stupid (I think
> that's known under the acronym KISS). I already argued about which choices
> I think are relevant for 90% of the transactions. Any further GUI elements
> that are relevant only to a minority of the transactions should absolutely
> be kept *out* of the big list.
>
> Again, if you want to implement it in a way so that for the usage from HBCI
> it can be switched off / made invisible, then I'm fine with anything. Also
> note that if my GUI concept just doesn't give you enough freedom to add
> more features, you are absolutely free to return to your initial
> Transaction-matcher GUI. Really, and that's not even too much of an effort
> duplication (since the gnc-gen-transaction code already uses 90% of the
> Transaction-matcher code). It would just acknowledge the fact that in HBCI,
> I can use yet more assumptions to get the simplest GUI possible, whereas in
> OFX, you see reasons that judge building a much more complicated and
> feature-rich GUI.

Considering the above and the fact that we are way too close to a release,  I 
think we no longer have a choice, we must fork.  I suggest that you swallow a 
copy of gnc-gen-transaction.c back into the HBCI module.  Considering I will 
keep your basic interface design, Transaction-matcher.c can hopefully stay 
common (the action menu will be moved out of that file).  I now compile with 
HBCI support, and I'll try to maintain compatibility as much as possible.  I 
still have the (perhaps arrogant) hope that you will come back to the generic 
matcher once it is completed.

This is unfortunate, but we no longer have time to argue this before release 
if we can't set true common objectives.  Email is just too time consuming (I 
already spend almost three hours on this one).  I've worked very hard to 
accommodate you until now, and I would be quite willing to continue to do so, 
especially since I've now seen what is to be gained by working together.  
Even with our different coding style I found your code cleanup amazing.

However, you just seem to be unwilling to compromise much.  I feel I already 
compromised a LOT (Building and debugging the new action handling and GUI 
took DAYS, and was mostly done to make you happy) and until now I mostly 
defended my arguments instead of attacking yours, but we are weeks away from 
final release, and I don't think I can afford to maintain this attitude.  But 
taking a confrontational approach towards you by email is unlikely to help us 
get ready in time for 1.8.

I am afraid a temporary fork might be the only viable solution.  We 
shouldn't have attempted this into a release cycle.  Neither of us have time 
time right now to finish a discussion on a generic interface by email.  

Even if we do fork, I'd be willing to make an oversea call if you send me your 
phone number by email.  That way we can save as much common code as we can, 
and perhaps understand each other better.

Good night,