SQL backend and dirty books

[Phil Longstaff]

Hi Derek,

I know you're not a fan of top posting, but that's how Yahoo mail does it.

I assume that the only meaning to "dirty" is "in the engine but not yet saved to permanent storage".

Part of my frustration is tripping over *magic* stuff that is not adequately documented.  The dirty/clean state of transactions and splits controls a lot of different activity during the transaction commit process, for example.  I'd love to see sequence diagrams for the whole transaction commit process (hmm, maybe I should create them).  Stuff happens in the background because there is a fairly intricate architecture.

On a slight tangent, the UI and engine don't have the concept of a group of objects needing to be modified together.  For example, if I scrub an account, I might create some lots, and modify some splits to refer to them.  In the database, creating a lot and modifying the splits for the lot should be done atomically.  Similarly, when a lot is deleted, clearing the split's lot_guid and deleting the lot should be done atomically, and in that order.  When deleting a transaction, the splits should be deleted first.  This is all in the name of 1) total data correctness, and 2) referential integrity.  Since sql databases have a transaction idea, perhaps there should be some sort of gnc_engine_begin_edit()/gnc_engine_rollback_edit()/gnc_engine_commit_edit() to control a group of changes in multiple objects which need to be kept in sync (suggest alternate routine names if you want).

Phil


----- Original Message ----
From: Derek Atkins <warlord at MIT.EDU>
To: Phil Longstaff <plongstaff at rogers.com>
Cc: GnuCash development list <gnucash-devel at gnucash.org>
Sent: Friday, July 11, 2008 9:43:42 PM
Subject: Re: SQL backend and dirty books

Phil,

Quoting Phil Longstaff <plongstaff at rogers.com>:

> I've investigated my transaction/split issue further and it appears 
> as though the problem is in how qof determines that the books are 
> dirty.
>
> There is an alt_dirty_mode flag in qof and the gc engine sets it to 
> TRUE.  When this flag is set, *after* the backend is called during 
> the second part of a qof_commit_edit() call, the collection and book 
> are marked dirty.  While this may be fine for an XML backend where 
> the commit does not really save anything and therefore the books 
> really *are* dirty, it is not OK for an SQL backend.  If the concept 
> of the books being dirty means that there is unsaved content, then 
> the backend should have more control over that status.
>
> Derek's vision seems to be that sqlite replaces xml as the standard 
> backend, and that xml might still exist as an import/export 
> mechanism.  If that is what is adopted (seems fine to me, but some 
> concensus would be good), then I'll need to look at and possible 
> clean up the whole dirty mechanism.

It would be nice if there were more consensus on that.  :)

> For now, I think I've fixed my problem by pushing the problem code 
> down into the xml backend and out of qof.  This means that a commit 
> using the file backend *does* mark the books as dirty, whereas a 
> commit using the sql backend does not.

I think it depends on how you did it.  I haven't looked at your changeset
yet, but how I think it should work is:

1) In the first part of the commit, qof (or above) sets the object dirty.
2) The backend is called.  The backend can clear the dirty flag if necessary.

One issue is that there are still ways to dirty the book outside of
a begin/commit.  In particular modifications to a KVP Frame don't
take a begin/commit and don't dirty the book.

I don't see why the book would be dirtied AFTER the commit.  You are
correct that THAT is just broken.

> Phil

-derek

PS: I'm sorry I haven't responded to BOTH of your queries on this topic.
I've been so busy with real work and some house construction that I've
dropped lots of things on the floor, this being one of them.  All I can
say is I'm Sorry, and I'll try to be a little more responsive going forward.
On the other hand, I start one month of travel in about 13 hours so I
dont know how responsive I'll be while on the road, either.  I'll do what
I can, and maybe respond from 35,000 feet.  ;)

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
      warlord at MIT.EDU                        PGP key available