[GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

Christian Gruber christian.gruber at posteo.de
Mon Nov 11 16:52:26 EST 2019

Am 10.11.19 um 23:28 schrieb Christian Gruber:
> Am 10.11.19 um 22:39 schrieb John Ralls:
>>> On Nov 10, 2019, at 12:07 PM, Christian Gruber 
>>> <christian.gruber at posteo.de> wrote:
>>> Am 08.11.19 um 23:08 schrieb John Ralls:
>>>>> On Nov 8, 2019, at 1:58 PM, Christian Gruber 
>>>>> <christian.gruber at posteo.de> wrote:
>>>>> Am 08.11.19 um 04:39 schrieb John Ralls:
>>>>>> Christian,
>>>>>> It's not that it's not prepared for Bayesian matching, it's that 
>>>>>> older versions of GnuCash stored the Bayesian match tokens 
>>>>>> hierarchically. Aaron Laws (lmat) changed it to a flatter 
>>>>>> structure with somewhat better memory locality for faster access. 
>>>>>> imap_convert_bayes_to_flat should run once to convert the data 
>>>>>> and set the feature, after which check_import_map_data will see 
>>>>>> the flag and return. A file created with 3.x and Baysian maps 
>>>>>> would already have the feature set.
>>>>>> With that background, to your questions:
>>>>>> Why does it take so long? Because it traverses the entire tree of 
>>>>>> accounts, every time. The test book has 1127 accounts. Add to 
>>>>>> that that there are some things inside the loop that shouldn't be 
>>>>>> and that convert_imap_account_bayes_to_flat doesn't use some 
>>>>>> obvious short circuits and you get taking a long time.
>>>>>> Why does it run twice? Because there aren't any accounts with 
>>>>>> import-map-bayes slots, so it does no conversions so it doesn't 
>>>>>> set the feature.
>>>>> Why isn't the feature set in any case after conversion is done, 
>>>>> whether there was any slot to convert or not?
>>>>> I would expect that, if the hierarchical structure should not be 
>>>>> used anymore. I wouldn't only set the feature, if there has been 
>>>>> any error during conversion. But it's not an error, if 
>>>>> convert_imap_account_bayes_to_flat() returns false.
>>>> The feature isn't set because that would prevent using the file 
>>>> with an older version of GnuCash for no good reason: There aren't 
>>>> any import-map-bayes tags in the new format so there's nothing for 
>>>> the older version to mess up.
>>>> Regards,
>>>> John Ralls
>>> Sounds meaningful, but it's not obvious. The conversion process is 
>>> not much transparent to the user. The user is not informed about 
>>> conversion and additionally the conditions, when the GnuCash file is 
>>> converted and when not, are not quite intuitive, I guess. And the 
>>> second question for me is, if this is really a relevant use case to 
>>> keep the GnuCash file unconverted if possible, when it is used with 
>>> a newer GnuCash version. Maybe it's relevant, if several people, who 
>>> use different GnuCash versions work on the same GnuCash file. But 
>>> what happens, when one with a newer GnuCash version uses the 
>>> bayesian import matcher for the first time, which silently converts 
>>> the GnuCash file? Will it still be usable with an older GnuCash 
>>> version?
>>> But actually this is another discussion, which is not important for 
>>> the bug ticket. What's more important, I created a new fresh GnuCash 
>>> file with the SKR04 accounts template using my GnuCash version 3.7 
>>> and if I checked this correctly, even with this GnuCash version the 
>>> feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash 
>>> file. Can you confirm this? If yes, then the bug report is relevant 
>>> not only for migration from older to current GnuCash versions but 
>>> even for current GnuCash versions in general.
>> No, it's the same discussion. If you don't use the feature then the 
>> flag for it won't be set. That's how we designed the feature 
>> mechanism. The use case is straightforward: Some user has two 
>> computers, a laptop running Windows and a desktop running Ubuntu 
>> 16.04. She updates the Windows machine with each release but doesn't 
>> know how to build GnuCash on her Ubuntu machine so leaves it at 
>> whatever they shipped on 16.04; 2.6.12 IIRC. As long as she doesn't 
>> use any of the features supported only in 3.x she can continue to use 
>> both systems to keep her books and regardless of which machine she 
>> uses to create a book.
>> The "silently" part could be better. It would indeed be a nice 
>> enhancement to pop a dialog box when GnuCash is about to set a 
>> feature flag explaining that it's use will preclude using the file on 
>> a GnuCash version older than X and giving the user a chance to bail 
>> out. It would be nicer still to persist that decision so that it need 
>> ask only once, and if the user assents it could set the feature flag 
>> even if the feature isn't used, as is the case here. Of course there 
>> would then be the need to reset negative responses so that when our 
>> hypothetical user upgrades her desktop to Ubuntu 20.04 next spring 
>> she can turn on all of those 3.x only features she'd previously 
>> declined.
>> It occurs to me that another improvement to the existing flat 
>> Bayesian conversion would be to check the state of the 
>> GNC_PREF_USE_BAYES and skip the conversion if it's not true.
> This is already implemented. Flag GNC_PREF_USE_BAYES is checked in 
> functions matchmap_find_destination() and matchmap_store_destination() 
> and if not set, than functions gnc_account_imap_find_account_bayes() 
> and gnc_account_imap_add_account_bayes() aren't invoked and no 
> conversion will be done.
>> Regards,
>> John Ralls
> Thanks for the revealing explanation of the feature mechanism. In this 
> case it would be really helpful, if GnuCash would explain itself and 
> interact with the user, when features are changed and conversions are 
> done, which are not backwards compatible. Moreover it would be 
> meaningful, if the user could "enable/disable" backwards compatibility 
> when creating a new GnuCash file to avoid such extensive conversion 
> steps. This means, that all supported features of the used GnuCash 
> version are set by default during creation. Alternatively, if it's 
> possible to significantly speed up searching for (non-existing) 
> bayesian match tokens, this should be preferred.
> I'll create an enhancement ticket.
> Regards,
> Christian
I added enhancement ticket 797499 

More information about the gnucash-devel mailing list