--add-price-quotes corrupted data file

[Derek Atkins]

David Carlson <david.carlson.417 at gmail.com> writes:

> On 4/21/2014 11:58 PM, KevinK wrote:
>> GnuCash 2.4.10 on Ubuntu 12.04
>> I've been using GnuCash since December 2004 and I have a tendency to keep
>> all my data unless there is some pressing need to purge.
>>
>> When I tried to open GnuCash today, it told me there was a lock.  Being
>> single user, I bypassed that and then GnuCash went into a 100% CPU loop on
>> the loading data splash screen.  The splash screen did not update at all and
>> even waiting 15 to 20 minutes did not change anything.  I killed the process
>> and tried several things, all with the same result.  I finally recovered by
>> deleting the gnudata file and renaming the immediate prior backup.
>>
>> I was able to determine the problem was caused by the --add-price-quotes
>> function.  I use a cron job to run "gnucash --add-price-quotes" every
>> weekday at a set time (many ticker symbols, all collecting from Yahoo).  The
>> bad gnudata file along with the lock file were timestamped at the daily
>> collection time on April 16th.  The backup file I recovered from was
>> timestamped at the daily collection time on April 15th (my last interactive
>> session was on April 14th).  When I looked in the price editor, the most
>> recent price quote for any stock was April 8th - so, the quote collection
>> ran on the 9th, 10th, 11th, 14th, and 15th without recording any stock
>> prices and without corrupting the data file.  However, the quote run on the
>> 16th left an unusable data file along with the lock file.
>>
>> After recovering from the backup, I deleted price history prior to 2013
>> (data file size dropped from 4.5Mb to 3.3Mb).  After saving my work, I was
>> able to retrieve quotes again with no apparent adverse effects.
>>
>> I no longer have a problem but wanted to post this in case someone else ran
>> across the same issue and needed an explanation.  Solution: restore a backup
>> and purge some price history.
>>
>> Developers may want to look at a more graceful way to tell users that they
>> are hoarding too much data.
>>
> It seems odd to me that the problem was caused by too much history, but
> perhaps so.  I would consider that to be a bug if true.
> Is it possible that there was another cause that happened to be fixed by
> selecting a backup?  Could there be a subtle bug in your cron job, for
> example?

Kevin, another thing you can do, especially if you have the data from
just before and just after the corruption is to perform a "diff" on the
file to figure out exactly what changed.  That might show you the
corruption.  Maybe there was a race condition?  Or it could be that
there are algorithms within GnuCash that just doesn't like that much
history?  It's hard to say exactly, but looking at the data file change
from "working" to "not working" may provide some insight.

You will possibly need to gunzip the files first before you run diff.

Good Luck,

> David C

> Please remember to CC this list on all your replies.
> You can do this by using Reply-To-List or Reply-All.

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord at MIT.EDU                        PGP key available