discussion generic import design

[Derek Atkins]

Christian Stimming <stimming@tuhh.de> writes:

> Ok, I see we are in fact getting somewhere. Sorry that I didn't state
> my intentions earlier. I now see that one reason why I'm unhappy with
> the design you decided on, is that eventually in fact the HBCI code
> needs something else than the OFX and other code. In HBCI, I can
> *always* rely on several assumptions that *greatly* simplify or
> streamline the importing process:

I still believe that we can have a 'streamlined' process while
re-using the same importer technology.  (Youcan have a druid that
skips pages when no work needs to be done).

> * The originating account for a transaction is always known
> * The transaction currency is always known
> * The HBCI importer can always specify only one split of the
> transaction (i.e. the originating one) [1]
> * Transaction matching always happens according to the
> memo/description/payee string
> * No transactions with interdependencies are imported, i.e. the
> protocol importer doesn't have transactions with both splits already
> specified but only with one split as mentioned above.
> * The transactions to be imported are already available in one
> specific list, i.e. the importing process has a well-defined beginning
> and end.
> 
> I thought (and think) that the qif-importer represents a GUI that
> works very well with these assumptions given. The reason why I raised
> all this fuzz is that I realized that you were heading into a
> direction of a much more general GUI, but as I just described, for
> HBCI this would be an over-complexification of the whole importing
> process.

The current qif-importer does _not_ know the originating account.  I
don't know if it knows the txn currency.  QIF can specify multiple
splits, and it _CAN_ specify the "other account", but usually it's
just a category (which is _like_ an account), and sometimes it matches
on memo/description/payee string.  So again, this does not completely
match the HBCI limitations.

One assumption that is 100% equal between HBCI and QIF is the last
one, that the transactions to be imported are already available in one
list.

> Conclusion so far: I think I will give the qif-importer GUI a try, but
> not because I don't like your code but rather because we now
> eventually realized that the requirements are in fact quite
> different. Therefore the generic importer is fine as it is, and should
> be further optimized for a generic (probably mostly file-based) import
> -- and as Derek already pointed out, it would be a great basis for
> import from CSV or XML or a whole lot of other account manager's file
> formats. But since HBCI comes with the very rigid restrictions
> mentioned above, a much more restricted GUI might do a better job
> there. Nevertheless in one week from now I might still realize that
> deriving from the qif-importer ends up way more borken and complex
> than ever before... then I might have to repent, tear my clothes and
> put ash on my head...

I also think the generic-import UI would be good for QIF.  I would
like to see a single generic-import interface that can be used for
EVERYTHING.  Perhaps certain specific methods can skip certain steps
(i.e., HBCI does not need to "pick the originating account").
However, having generic, reusable UI pieces for these functions would
be a Good Thing (TM).

FTR, I do NOT think the existing qif-import druid would be easy for
you to use for HBCI.  Feel free to take a look, but I think it is way
too tied to the qif importer, and it would be more work to rip that
apart then it would be to extend Benoit's generic-importer UI to
handle HBCI and skip the unnecessary steps.

>  >>struct ImportTrans {
>  >>  /* gnucash transaction with everything filled and with one split, but
>  >>     which was not yet inserted into the account */
>  >>  Transaction *trans;
>  >
>  > You can't, according to Transaction.h, a split MUST be parented by
> an account
>  > before you use any of the xaccSplitSetAmount() and similar
> functions on it.
>  > And since an imported transaction could have any number of splits,
> the splits
>  > must be set.
> 
> Oops, that's right. But: For HBCI, again I know that I always have
> only one split.
> 
> But in general: It would still be possible for the generic importer to
> not only define its own transaction data type but also its own split
> data type. This is precisely what the qif-importer does (with types
> defined in qif-objects.scm).

No, because as soon as you xaccSplitAddToAccount() the split will show
up in the AccountRegister.  This is why the old qif-importer _AND_ the
new qif-importer create a new AccountGroup that mirrors the real
AccountGroup, adds the imported transactions to the new AccountGroup,
and then either merges or destroys the new AccountGroup at the end.

> struct ImportTrans {
>    Transaction *trans;
>    GList *split_list;
>    char *payee_memo;
> }
> struct ImportSplit {
>    char *memo;
>    gnc_numeric amount;
>    Account *account;
> }
> 
> or similar. Then the importer will create the "real" split for each
> ImportSplit, add it to the respective account, fill in the amount, add
> it to the transaction etc. etc.

As I said, if the Split->Account points to a "real" account, it will
show up in the register.  This is not what you want before the import
is completed.

> As for the lookup table of payee/memo matching: The qif-importer
> implements it in a scheme "alist" which is stored from session to
> session in a seperate file in $HOME/.gnucash/qif-map-something
> . Whether a seperate file or some KVP would be the better place to
> store it --- I don't really care, since I think it doesn't really make
> much of a difference. So that's just an implementation question.

There are a number of ways to do this, the scheme alist is just one.
We can (SHOULD) create a generic mapper interface so the various
importers can use them.  It is clear that OFX and QIF need the maps.

> Christian

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord@MIT.EDU                        PGP key available