[GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs
christian.gruber at posteo.de
Mon Nov 11 16:52:26 EST 2019
Am 10.11.19 um 23:28 schrieb Christian Gruber:
> Am 10.11.19 um 22:39 schrieb John Ralls:
>>> On Nov 10, 2019, at 12:07 PM, Christian Gruber
>>> <christian.gruber at posteo.de> wrote:
>>> Am 08.11.19 um 23:08 schrieb John Ralls:
>>>>> On Nov 8, 2019, at 1:58 PM, Christian Gruber
>>>>> <christian.gruber at posteo.de> wrote:
>>>>> Am 08.11.19 um 04:39 schrieb John Ralls:
>>>>>> It's not that it's not prepared for Bayesian matching, it's that
>>>>>> older versions of GnuCash stored the Bayesian match tokens
>>>>>> hierarchically. Aaron Laws (lmat) changed it to a flatter
>>>>>> structure with somewhat better memory locality for faster access.
>>>>>> imap_convert_bayes_to_flat should run once to convert the data
>>>>>> and set the feature, after which check_import_map_data will see
>>>>>> the flag and return. A file created with 3.x and Baysian maps
>>>>>> would already have the feature set.
>>>>>> With that background, to your questions:
>>>>>> Why does it take so long? Because it traverses the entire tree of
>>>>>> accounts, every time. The test book has 1127 accounts. Add to
>>>>>> that that there are some things inside the loop that shouldn't be
>>>>>> and that convert_imap_account_bayes_to_flat doesn't use some
>>>>>> obvious short circuits and you get taking a long time.
>>>>>> Why does it run twice? Because there aren't any accounts with
>>>>>> import-map-bayes slots, so it does no conversions so it doesn't
>>>>>> set the feature.
>>>>> Why isn't the feature set in any case after conversion is done,
>>>>> whether there was any slot to convert or not?
>>>>> I would expect that, if the hierarchical structure should not be
>>>>> used anymore. I wouldn't only set the feature, if there has been
>>>>> any error during conversion. But it's not an error, if
>>>>> convert_imap_account_bayes_to_flat() returns false.
>>>> The feature isn't set because that would prevent using the file
>>>> with an older version of GnuCash for no good reason: There aren't
>>>> any import-map-bayes tags in the new format so there's nothing for
>>>> the older version to mess up.
>>>> John Ralls
>>> Sounds meaningful, but it's not obvious. The conversion process is
>>> not much transparent to the user. The user is not informed about
>>> conversion and additionally the conditions, when the GnuCash file is
>>> converted and when not, are not quite intuitive, I guess. And the
>>> second question for me is, if this is really a relevant use case to
>>> keep the GnuCash file unconverted if possible, when it is used with
>>> a newer GnuCash version. Maybe it's relevant, if several people, who
>>> use different GnuCash versions work on the same GnuCash file. But
>>> what happens, when one with a newer GnuCash version uses the
>>> bayesian import matcher for the first time, which silently converts
>>> the GnuCash file? Will it still be usable with an older GnuCash
>>> But actually this is another discussion, which is not important for
>>> the bug ticket. What's more important, I created a new fresh GnuCash
>>> file with the SKR04 accounts template using my GnuCash version 3.7
>>> and if I checked this correctly, even with this GnuCash version the
>>> feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash
>>> file. Can you confirm this? If yes, then the bug report is relevant
>>> not only for migration from older to current GnuCash versions but
>>> even for current GnuCash versions in general.
>> No, it's the same discussion. If you don't use the feature then the
>> flag for it won't be set. That's how we designed the feature
>> mechanism. The use case is straightforward: Some user has two
>> computers, a laptop running Windows and a desktop running Ubuntu
>> 16.04. She updates the Windows machine with each release but doesn't
>> know how to build GnuCash on her Ubuntu machine so leaves it at
>> whatever they shipped on 16.04; 2.6.12 IIRC. As long as she doesn't
>> use any of the features supported only in 3.x she can continue to use
>> both systems to keep her books and regardless of which machine she
>> uses to create a book.
>> The "silently" part could be better. It would indeed be a nice
>> enhancement to pop a dialog box when GnuCash is about to set a
>> feature flag explaining that it's use will preclude using the file on
>> a GnuCash version older than X and giving the user a chance to bail
>> out. It would be nicer still to persist that decision so that it need
>> ask only once, and if the user assents it could set the feature flag
>> even if the feature isn't used, as is the case here. Of course there
>> would then be the need to reset negative responses so that when our
>> hypothetical user upgrades her desktop to Ubuntu 20.04 next spring
>> she can turn on all of those 3.x only features she'd previously
>> It occurs to me that another improvement to the existing flat
>> Bayesian conversion would be to check the state of the
>> GNC_PREF_USE_BAYES and skip the conversion if it's not true.
> This is already implemented. Flag GNC_PREF_USE_BAYES is checked in
> functions matchmap_find_destination() and matchmap_store_destination()
> and if not set, than functions gnc_account_imap_find_account_bayes()
> and gnc_account_imap_add_account_bayes() aren't invoked and no
> conversion will be done.
>> John Ralls
> Thanks for the revealing explanation of the feature mechanism. In this
> case it would be really helpful, if GnuCash would explain itself and
> interact with the user, when features are changed and conversions are
> done, which are not backwards compatible. Moreover it would be
> meaningful, if the user could "enable/disable" backwards compatibility
> when creating a new GnuCash file to avoid such extensive conversion
> steps. This means, that all supported features of the used GnuCash
> version are set by default during creation. Alternatively, if it's
> possible to significantly speed up searching for (non-existing)
> bayesian match tokens, this should be preferred.
> I'll create an enhancement ticket.
I added enhancement ticket 797499
More information about the gnucash-devel