QSF XML file backend for gnucash-gnome2-dev branch

Tue Jan 25 17:45:31 EST 2005

On Tuesday 25 January 2005 9:18 pm, Derek Atkins wrote:

Ignore that one about balance sheets - I had too many email windows open. 
Sorry.

> Neil,
>
> First, I'd like to say Nice Job!

Thanks! I wasted a lot of time last year thinking around the problem of 
mapping objects between QOF applications - thinking of scheme and lua and oaf 
and others. Suddenly, over the holidays, it all came together in XML and I 
just wanted to get some of the code released so that I can concentrate on the 
next stages.

> I know you've put a lot of hard work 
> into this, so I hope my comments below don't discourage you...

No, but the bug I've just found might!

:-(

QOF_TYPE_INT64 and QOF_TYPE_NUMERIC support aren't as robust as I intended, 
but I can fix it.

It's human nature, I suppose, you always find a bug AFTER the release!

> Also, 
> I should note that I haven't looked at the patch -- I'm basing my
> comments below on your particular statements.
>
> Neil Williams <linux at codehelp.co.uk> writes:
> > The patch includes the QSF schema for object and map files, it includes
> > code to install the schema in a qsf sub-directory of $prefix/share/ and
> > it also includes a test QSF map that will be the next phase of
> > development to map pilot-link QOF objects into GnuCash objects.
>
> I'm wondering if these shouldn't go into $prefix/share/xml/gnucash/...
> (or .../share/xml/qof).  You don't say specifically where under
> $prefix/share you're putting them.

Sorry - there's so much in the patch I didn't know what to leave out in the 
email. 

At present, if prefix is given to autogen.sh as:
--prefix=/opt/garfield/qof
the schemas get installed to:
/opt/garfield/qof/share/qsf/
for QOF.

In GnuCash, for a prefix of:
--prefix=/opt/garfield/gnucash
the schemas are installed into:
/opt/garfield/gnucash/share/qsf/

On a standard package install, that would be:
/usr/share/gnucash/share/qsf
(other directories in gnucash/ being bin/ lib/ etc.)

If you want that to be:
/opt/garfield/gnucash/share/xml/qsf/
and
/usr/share/gnucash/share/xml/qsf/

That's easy. I used qsf/ to distinguish from any other xml formats.

>
> > Importing data will be done via a second QofSession, loading the QSF XML
> > using qof_session_load which will automatically call qof_book_merge to
> > control the merge into the existing QofBook in the original QofSession.
> > I'll add File menu options for this in due course.
>
> What happens if I just run File -> Open and select of QSF file?
> it "automatically call qof_book_merge"?  I think the "load" and
> "merge" apis should definitely be separable.

The QSF usually contains only a partial book and certainly doesn't have to 
contain AccountGroup so it can't be loaded as the main GnuCash book - nothing 
will show up in the CoA in most cases. It needs to be merged into an existing 
book somehow.

So, you can't really load QSF directly (at least, not always) - on Open, the 
final code can inform the user that the QSF will be merged into the current 
QofBook, giving the option to abort.

I will sort out the File -> Import menu later. I just figured that someone, 
sometime, is going to try and open the file directly with File -> Open and I 
ought to make sure it identified it correctly. Plus it was easier to test it 
there than to add a whole new menu option and all the code that goes with it.

> > QSF source files are included in src/backend/file and create their own
> > library, gnucash/libqsf-backend-file.so
>
> I think the code should probably go into src/backend/qsf instead of
> backend/file.  Also, the library should probably be called
> libgnc-backend-qsf (or libqof-backend-qsf).

OK. It's easily separated out. I put it under file because the access method 
is file:/ and I followed Linas' naming of his DWI backend.

I can make it backend/qsf and then I can still have it in the same folders in 
QOF and GnuCash.

libqof-backend-qsf is preferable and I can easily switch to that.

> > The patch also includes error codes and error messages related to QSF
> > validation and parsing.
>
> Where?

qof_backend.h contains the error codes, added into the enum.

app-file/gnc-file.c contains the full error messages, corresponding to the 
macros from the enum. Other applications using QOF would need to define their 
own error messages to go with the enum, that's expected.

> > QSF increases the libxml2 requirement to 2.6.0 for the gnome2 branch to
> > support the schema validation which identifies QSF files relative to
> > existing files.
>
> This is a major issue.  Our target platform has libxml2-2.5.10, not
> 2.6.  We cannot depend on 2.6.

I thought it was OK to go to 2.5.2?  I just thought I'd play safe by going for 
2.6.

https://lists.gnucash.org/pipermail/gnucash-devel/2005-January/012511.html
> > FWIW, requiring libxml2 >= 2.5.2 shouldn't be a problem.  Even RH9
> > (updates) distributes 2.5.4, and FC1 uses 2.5.11 by default.  So I
> > don't see this as a problem.

I can go back to 2.5.2 but schemas simply weren't supported before that.

There may be a few problems at 2.5.2 - schemas were very new - but it should 
be OK - I'm not making huge demands on the schema code, at least not until I 
get to grips with KVP.

> > QSF uses UTC time throughout and uses this time format string:
> > #define QSF_XSD_TIME   "%Y-%m-%dT%H:%M:%SZ"
>
> Is there any particular reason you didn't use the existing Timespec
> XML encoding?

Won't validate as an XML date. QSF_XSD_TIME takes it's name from xsd: which 
determines the format. I lose a lot of validation checking if the date has to 
be a string type tag.

> > QOF_TYPE_INT32 is not a problem, QOF_TYPE_KVP is a little more of a
> > challenge.
>
> Why is QOF_TYPE_KVP a challenge?

Maps. I knew this would be the hardest part and it's not disappointing me.
:-(

Trying to calculate KVP frames from other object data is not my idea of fun. 
If I can handle them as defaults, booleans or some other lookup method, I 
will. I'd rather have 6 user defaults than a 6 level tree within the QSF. It 
makes validation a pig and it makes mapping v.complex.

(And this is whilst working in XML and C - imagine the nightmare if I'd gone 
for lua or oaf which are languages I'd have to LEARN before coding!!!!)

The disadvantage at this stage (which will be an advantage in the final code) 
is that the work now has to done within the QOF application instead of 
handing off to a separate routine when I was considering writing the map code 
in lua or oaf. It allows us to have online communication between QOF 
applications (a major goal), but the cost is the simple matter of programming 
by yours truly.

:-(

> KVPs are used a lot in gnucash (and 
> in particular in the business feature integration with the rest of
> gnucash).

I know (although I sometimes wish it didn't).

>
> Nice work so far,

This ain't the half of it!

I'm still hoping to get QOF built onto pilot-link for their next release and 
this will probably be before GnuCash for Gnome2 is released. The pilot-link 
code is a lot closer because it doesn't need any GUI code - a blessed relief 
for someone like me who has never been good at GUI programming - and it will 
allow them to have a user-editable off-line storage format for the first 
time. Current backups are binary copies of Palm databases - not that handy - 
or various, incomplete, exports in mutually incompatible CSV formats.

So, I commit to QOF regularly (can't get hold of Linas so he doesn't know yet 
- I've been trying off and on since September!), I've sent this patch for g2, 
I've got a patch for GnuCash HEAD upcoming and I've got the pilot-link code 
getting bigger by the day. (I can't send that patch in, they're up to their 
eyes in the 0.12 release. I'm hoping to get QOF into 0.13).

-- 

Neil Williams
=============
http://www.dcglug.org.uk/
http://www.nosoftwarepatents.com/
http://sourceforge.net/projects/isbnsearch/
http://www.neil.williamsleesmill.me.uk/
http://www.biglumber.com/x/web?qs=0x8801094A28BCB3E3

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.gnucash.org/pipermail/gnucash-devel/attachments/20050125/5c1421a5/attachment.bin