Performance improvement for xml loads (+comments)

Bill Carlson wwc@wwcnet.nu
Tue, 5 Dec 2000 00:41:53 -0500


Hi,

 	I've been trying to make the xml stuff go a bit
faster.  The following patch will cut my large file load
(~10000 transactions) from about 2 minutes to about 30 seconds.
This is obviously good (and along the lines of a change
I made a while ago for the binary file load).  

	The only problem I have now is that it is still
MUCH slower than the binary.  The file size is about 6x
the size before (9 Meg vs. 1.5 Meg) and to actually do
the write it seems to use about 50 Meg of ram, because
xml builds the whole tree in memory before writing anything.
This all concerns me a great deal as xml seems to be
the future for gnucash and this experience tells me
that we are in trouble as far as even moderatly large
databases of transactions.  Any comments?  I'd be more than
happy to help make things better.  One thought I had was
to try a "io-gncsql-r.c" and "io-gncsql-w.c" which would
talk to an sql database (this would be different than using
the database as the engine, it would just read and write
like a file).  Will the architecture of gnucash allow
for user preferences in the style of saving?

	In any case, what follows is my brief patch which
I'd appreciate if it were put into the CVS.  Thanks!

		Cheers,

		Bill Carlson
		wwc@wwcnet.nu


Index: src/engine/Group.c
===================================================================
RCS file: /home/cvs/cvsroot/gnucash/src/engine/Group.c,v
retrieving revision 1.61
diff -r1.61 Group.c
138a139,155
> xaccAccountGroupCommitEdit( AccountGroup *grp )
> {
>   int i;
> 
>   if (NULL == grp) return;
> 
>   for(i = 0; i < grp->numAcc; i++ )
>   {
>     xaccAccountCommitEdit(grp->account[i]);
>     xaccAccountGroupCommitEdit (grp->account[i]->children);
>   }
> }
> 
> /********************************************************************\
> \********************************************************************/
> 
> void
Index: src/engine/Group.h
===================================================================
RCS file: /home/cvs/cvsroot/gnucash/src/engine/Group.h,v
retrieving revision 1.34
diff -r1.34 Group.h
39a40
> void 	      xaccAccountGroupCommitEdit( AccountGroup *grp );
Index: src/engine/io-gncxml-r.c
===================================================================
RCS file: /home/cvs/cvsroot/gnucash/src/engine/io-gncxml-r.c,v
retrieving revision 1.5
diff -r1.5 io-gncxml-r.c
2217a2218,2222
>   /* commit all groups, this completes the BeginEdit started when the
>    * account_end_handler finished reading the account.
>    */
>   xaccAccountGroupCommitEdit (ag);
> 
2599a2605,2609
>   /* Now return the account to the "edit" state.  At the end of reading
>    * all the transactions, we will Commit.  This replaces #splits
>    *  rebalances with #accounts rebalances at the end.  A BIG win!
>    */
>   xaccAccountBeginEdit(acc);