discussion generic import design

[Christian Stimming]

Ok, I see we are in fact getting somewhere. Sorry that I didn't state my 
intentions earlier. I now see that one reason why I'm unhappy with the 
design you decided on, is that eventually in fact the HBCI code needs 
something else than the OFX and other code. In HBCI, I can *always* rely 
on several assumptions that *greatly* simplify or streamline the 
importing process:

* The originating account for a transaction is always known
* The transaction currency is always known
* The HBCI importer can always specify only one split of the transaction 
(i.e. the originating one) [1]
* Transaction matching always happens according to the 
memo/description/payee string
* No transactions with interdependencies are imported, i.e. the protocol 
importer doesn't have transactions with both splits already specified 
but only with one split as mentioned above.
* The transactions to be imported are already available in one specific 
list, i.e. the importing process has a well-defined beginning and end.

I thought (and think) that the qif-importer represents a GUI that works 
very well with these assumptions given. The reason why I raised all this 
fuzz is that I realized that you were heading into a direction of a much 
more general GUI, but as I just described, for HBCI this would be an 
over-complexification of the whole importing process.

Conclusion so far: I think I will give the qif-importer GUI a try, but 
not because I don't like your code but rather because we now eventually 
realized that the requirements are in fact quite different. Therefore 
the generic importer is fine as it is, and should be further optimized 
for a generic (probably mostly file-based) import -- and as Derek 
already pointed out, it would be a great basis for import from CSV or 
XML or a whole lot of other account manager's file formats. But since 
HBCI comes with the very rigid restrictions mentioned above, a much more 
restricted GUI might do a better job there. Nevertheless in one week 
from now I might still realize that deriving from the qif-importer ends 
up way more borken and complex than ever before... then I might have to 
repent, tear my clothes and put ash on my head...

Some comments on the technical details below.

[1] There's one exception: If a user transfers money between several of 
his own HBCI-enabled accounts, then the HBCI importer might realize this 
fact, and then it can specify both splits. However, this is really a 
very rare exception, and also it could just be found by the "normal" 
transaction matching, since that will most likely take the "destination 
account code" into account for matching.

Benoit wrote:
 > 1-(Not directly related to this particular issue, but a prerequisite):
 > Generate a preference page, in which we can check in what options we want
 > enabled (Currently the options are ADD, REPLACE, IGNORE, RECONCILE). 
  That
 > page should also contain:
 > - -ADD and RECONCILE match threshold (two integers)
 > - -Wether by default transaction should be ADDed or RECONCILEd with c 
(cleared)
 > or y (reconciled).  Someone who reconciles with a paper statement at 
the end
 > of the month probably wants c, while someone who uses the matcher to 
add all
 > his transaction frequently probably wants y.  We could also make ADD and
 > RECONCILE independent.

I appreciate the proposed compromise, and at this point in time this 
would be something that everbody could live with. However, have you 
followed the "general preference" discussion during the last days? 
They've argued about whether user-editable preferences should be created 
in the first place or not, and that creation of preferences should be 
avoided if they just cover a not-really-completed design. IMHO These 
preferences would unfortunately fit into exactly that category: Two 
integers for the Match Threshold may be fine for you to fine-tune the 
behaviour, but the ordinary user would just have Zero knowledge about 
what that integer should be. Maybe the user would be far better off if 
he wouldn't even now that the threshold can be varied. Maybe the 
threshold should only be varied between three text choices, "rigid, 
loose, very loose" or similar. Maybe the generic importer can even put 
some more magic into that process and e.g. remember the user's choices 
as "average matching value that the user accepted", and use a function 
of that as match threshold. I'm just throwing ideas out here, these 
don't have to implemented -- I just want to underline that creating a 
user-visible preference here might actually not be a favor to the user.

 > 3-The cell containing the action is text, so it isn't obvious at all 
that you
 > can take an action there.  A pixmap representing some sort of button 
must be
 > created for each action.  The design should ideally make it clear 
that this
 > button iterates over a list of choices (which we unfortunately can't
 > display).  Basically this is a spin button that spins only one way.

I totally agree that a cell containing text isn't clear enough about 
that this can do some action. Obviously you know better than me about 
what is possible in the Clist. Creating button-looking pixmaps as a 
workaround might do, but: do these pixmaps contain text? If yes, the 
translators would need to translate the pixmaps as well, which is not so 
  good. Maybe your current solution with a Box of Buttons next to the 
transaction list wasn't so bad after all. :-) Maybe it will be more 
intuitive if the buttons are somewhere grouped with the right 
split-list, since that too changes from transaction to transaction.

 >>There's one more issue with the generic importer, namely, the imported
 >>transactions are already created in the final accounts currently and they
 >>are already visible in that account's registers.
 >
 > I agree seeing them in the register below is a nuisance.
 > It a minor problem for OFX, but I think in your case you have the 
register
 > open for the account being imported, so it must be a real nuisance.

Absolutely right. Sorry that I didn't state that earlier.

 >>avoid that. Instead, I would suggest that the generic importer should
 >>define its own "transaction" data type, which might be something like
 >>
 >>struct ImportTrans {
 >>  /* gnucash transaction with everything filled and with one split, but
 >>     which was not yet inserted into the account */
 >>  Transaction *trans;
 >
 > You can't, according to Transaction.h, a split MUST be parented by an 
account
 > before you use any of the xaccSplitSetAmount() and similar functions 
on it.
 > And since an imported transaction could have any number of splits, 
the splits
 > must be set.

Oops, that's right. But: For HBCI, again I know that I always have only 
one split.

But in general: It would still be possible for the generic importer to 
not only define its own transaction data type but also its own split 
data type. This is precisely what the qif-importer does (with types 
defined in qif-objects.scm).

struct ImportTrans {
   Transaction *trans;
   GList *split_list;
   char *payee_memo;
}
struct ImportSplit {
   char *memo;
   gnc_numeric amount;
   Account *account;
}

or similar. Then the importer will create the "real" split for each 
ImportSplit, add it to the respective account, fill in the amount, add 
it to the transaction etc. etc.

As for the lookup table of payee/memo matching: The qif-importer 
implements it in a scheme "alist" which is stored from session to 
session in a seperate file in $HOME/.gnucash/qif-map-something . Whether 
a seperate file or some KVP would be the better place to store it --- I 
don't really care, since I think it doesn't really make much of a 
difference. So that's just an implementation question.

Christian