Module system: last call for objections
Bill Gribble
grib@billgribble.com
Wed, 1 Aug 2001 19:01:57 -0500
Hi all. I've been working on a module/plugin system for gnucash, as I
have reported on the lists before, and it's pretty much to the point
that I want to commit it so that I'm not constantly trying to manually
track changes to CVS (there is a significant amount of source code
movement in my source tree). At this point, there's not very much
actual value added to gnucash, but I promise there will be in the next
few weeks.
The purpose of this mail is to ask again for comments on the module
system I'm proposing, to give everybody fair warning about the massive
changes they will see on a 'CVS update', and to try to make sure that
if I have made a boneheaded mistake of some kind that we catch it now.
Any feedback is appreciated. I will wait until Friday morning to make
my commit of the changes described here to give people on list digests
to have a look before the fact.
Thanks,
b.g.
What does this mean?
--------------------
Let's address the most FAQ first. What is this all about?
There are several things that are happening at once here. I have made
a "module" or "plugin" system for Gnucash; I have moved many pieces of
core Gnucash functionality into modules; I have made plans to do more
reorganization and modification of gnucash in the near future to
support the modularization.
What's the difference between modules and plugins? I think of
"plugins" as being strictly optional functionality. In my mind,
plugins are things like extra DSP algorithms in a sound editor
program, or extra filters in the Gimp. I think we should provide that
kind of thing for Gnucash... if you want to load up some new reports
that you grab off the Net, for example, that's a plugin, and Gnucash
should let you load it up at run time without modifying the Gnucash
source code.
On the other hand, "modules" are core functionality that can be
separated from other core functionality. In Gnucash, there are lots
of big and interesting pieces that are glued together to make an
application: the engine, the register, the various backends, the
report system, and more. Right now, it's difficult to load up a subset
of the whole shebang.
For instance, it should be possible to write a simple Guile script to
import a QIF file and stuff it into your Gnucash accounts. At the
moment, the mechanics of the QIF importer are so tied up with the GUI
that this is not possible. If the actual QIF handling were a separate
"module" from the QIF import GUI (with suitable dependency
relationships) it would be possible to load just the import guts, the
gnucash engine, and the XML backend and do the import without getting
a GUI involved at all.
As another example, the RPC backend requires a "server" gnucash as
well as a "client" gnucash. Why should the "server" gnucash load the
register or any other GUI components at all? No reason, except that
right now you can't build Gnucash without bringing in the whole thing.
There are plenty of other reasons you might want a "headless" GUI-free
gnucash for ecommerce or multiuser applications.
The point of the module system is to (1) allow people to contribute
plugins or core functionality that can be used without modifying the
Gnucash source code, and (2) to allow the core functionality of
Gnucash to be broken up into manageable pieces that can be used
independently.
How does it work?
-----------------
Here's the file that will be in src/gnc-module/doc/design.txt.
=========================================================================
What is a Gnucash module?
-------------------------
A gnucash module is a dynamically loadable libtool library that
defines the following symbols:
/* what version of the module system interface is assumed */
int gnc_module_system_interface;
/* information about the module's version */
int gnc_module_current;
int gnc_module_revision;
int gnc_module_age;
/* init is called the first time the module is loaded */
int gnc_module_init(void)
/* on_load is called every time the module is loaded */
void gnc_module_on_load(void);
/* on_unload is called every time the module is unloaded */
void gnc_module_on_unload(void);
/* on_finish is called when the last reference is unloaded */
void gnc_module_finish(void)
/* descriptive information */
char * gnc_module_path(void);
char * gnc_module_description(void);
gnc_module_system_interface is the revision number of the interface
listed above (i.e. the names and type signatures of the symbols each
module must define). The current revision number is 0; all modules
should assign gnc_module_system_interface = 0. As the interface
evolves, this should allow us to continue to load older modules.
The current, revision, age triplet describe the module version
according to libtool(1) conventions. To quote from the libtool info
pages:
1. Start with version information of `0:0:0' for each libtool library.
2. Update the version information only immediately before a public
release of your software. More frequent updates are unnecessary,
and only guarantee that the current interface number gets larger
faster.
3. If the library source code has changed at all since the last
update, then increment REVISION (`C:R:A' becomes `C:r+1:A').
4. If any interfaces have been added, removed, or changed since the
last update, increment CURRENT, and set REVISION to 0.
5. If any interfaces have been added since the last public release,
then increment AGE.
6. If any interfaces have been removed since the last public release,
then set AGE to 0.
gnc_module_path should return a newly-allocated string containing the
"module path". The module path is a logical path with elements
separated by /. The path will be used to structure views of available
modules so that similar modules can be grouped together; it may or may
not actually correspond to a filesystem path. The last element of the
module path is the name of this module. For example,
char * path = "gnucash/engine";
defines the "engine" module, which is in the "gnucash" group of
modules.
gnc_module_description should return a newly-allocated 1-line
description of what the module does. This can be displayed by GUI
elements to allow users to select modules to load.
Initializing the module system
------------------------------
Somewhere at program startup time, you need to call
gnc_module_system_init from C (see below from Scheme). This scans the
directories in the GNC_MODULE_PATH and builds a database of the
available modules. If you don't expliitly call
gnc_module_system_init, it will be called automatically the first time
you call gnc_module_load. Since there is some overhead in scanning
the installed plugins, you should probably call it explicitly at
program start time so you know when this overhead will happen.
In Scheme, you need to (use-modules (gnucash gnc-module)) and call
(gnc:module-system-init) if it was not called from C. You will need
to 'use-modules' this module if you intend to use any module system
functions from Scheme.
On the Scheme side, gnc:module-system-init is not a g-wrapped
function, because the gnc_module library's g-wrapped function bindings
are not published until *after* you call gnc_module_system_init. It
uses Guile's dynamic-link to open the libgncmodule.la library directly
and call gnc_module_system_init. This requires 'libgncmodule.la' to
be in the application's LD_LIBRARY_PATH. gnc:module-load is also a
'special' non-g-wrapped function that you can call immediately after
(use-modules (gnucash gnc-module)).
You can rebuild the module database at any time (say, if you know a
new module has been installed or the user has changed the module path
via some in-program mechanism) by calling gnc_module_system_refresh.
Loading modules
---------------
>From C call gnc_module_load(path, interface), or gnc:module-load from
Scheme. This returns a GNCModule (<gnc:module>) if a qualifying
module was successfully loaded, #f / FALSE otherside. GNCModule is an
opaque type.
A qualifying module is any module whose gnc_module_path matches the
path specification and for whom "interface" (an integer) falls between
gnc_module_current and (gnc_module_current - gnc_module_age). If
multiple modules qualify, libtool's rules are used to determine which
to load: the larger of gnc_module_interface, gnc_module_age, and
gnc_module_revision take precedence, in that order.
Module initialization
---------------------
The first time a module is loaded, its gnc_module_init function is
called. Any startup/initialization code should be defined in this
function. If this module depends on other modules, put the necessary
gnc_module_load calls in the init function.
If gnc_module_init returns FALSE, the module is not loaded. Any
cleanup must be done within gnc_module_init before returning.
Every time the module is loaded, including the first time, its
gnc_module_on_load function is called. The first time a module is
loaded, gnc_module_init is called first, then gnc_module_on_load.
If you have g-wrapped or other bindings that need to get published
when the module is loaded, put the (use-modules) calls in C in the
on_load function. This will ensure that bindings appear in the
current module if you are making the call from the Scheme side.
Module finalization
-------------------
Each time gnc_module_unload (gnc:module-unload) is called on a module,
any function that the module defines called 'gnc_module_on_unload' is
called.
When a module's reference count drops to 0 (i.e., when every module
that has called gnc_module_load on it has subsequently called
gnc_module_unload), the module's function gnc_module_finish is called,
and the library is dlclose()d.
=====================================================================
What are the licensing issues?
------------------------------
This is probably up for debate, and i'm no expert, but I am working
under the strongest GPL assumption: any gnucash plugin/module is a
derivative work of Gnucash, even if distributed separately. Therefore
any module that is distributed must be distributed under the GPL.
What are the changes to the source tree?
----------------------------------------
In my source tree, there are the following modules (left col is the
subdirectory of src/ and also module name).
gnc-module The module system (actually not a module)
engine The engine module
backend/file Binary and XML (v1 and v2) modules
backend/postgres Postgres backend
backend/rpc RPC backend
register/ledger-core The xacc SplitLedger and MultiLedger parts formerly
in src/
register/register-core Toolkit independent register code, formerly in
src/register/
register/register-gnome Gnome-specific register code, formerly in
src/register/gnome
import-export/qif-import the old qif importer with Gnome druid
The code from src/engine is now in src/engine/engine, except for the
code related to the file backend, which is in src/backend/file ; the
code from src/engine/sql is now in src/backend/postgres ; the code
from src/engine/rpc is in src/backend/rpc.
g-wrap bindings from src/guile/gnc.gwp that were wrapping engine
functions are now in src/engine/gw-engine-spec.scm; ones related to
glib are in src/engine/gw-glib-spec.scm, but should move to a separate
glib module, most likely. Helper functions related to engine g-wrap
bindings have been moved from src/guile/gnc-helpers.c to
src/engine/engine-helpers.c.
The register code from src/ (MultiLedger and SplitLedger) is now in
src/register/ledger-core. Code from src/register is in
src/register/register-core. Code from src/register/gnome is in
src/register/register-gnome.
The Gnome and Guile code related to the current QIF importer is in
src/import-export/qif-import.
How do I use modules?
---------------------
Right now, the module system is sort of glued on to Gnucash. I have
modularized the engine, the various backends, the register, and the
QIF importer. However, since the code that uses these various
features in the Gnucash app was not designed to work with a module
system, there have been compromises, mainly in that the Gnucash app is
linked against the various modules at build time rather than
dynamically loading them at run time.
For example, the Gnucash application is linked against the QIF
importer module, even though the API is just one function that could
be dynamically looked up. This means that the Scheme code for
importing QIF files, and the Gnome code for the QIF import GUI, is
loaded at application startup time, whether or not you ever use them.
To me, it's preferable to shorten the app startup time by dynamically
loading the Gnome and Scheme code the first time the user tries to run
the QIF importer.
This could be fixed by having the "load a QIF" callback in the Gnucash app
be rewritten to do this:
GNCModule qif = gnc_module_load("gnucash/import-export/qif-import", 0);
if(qif)
{
void (* launch_importer)(void) =
gnc_module_lookup(qif, "gnc_qif_import_druid_create");
launch_importer();
}
Or, even better, by using Scheme for the button callback. On the Scheme
side, g-wrapped function bindings are automatically published by a
gnc:module-load, so:
(if (gnc:module-load "gnucash/import-export/qif-import" 0)
(gnc:qif-import-druid-create))
Dynamic loading of modules will make the Gnucash startup process much
faster. As a small but significant bonus, it could make the splash
screen appear much sooner too: we could easily just load the splash
screen module before doing anything else, and have it visible while
loading all the other modules necessary to start the gnucash app.
I'd eventually like the Gnucash startup process to be a simple Scheme
file without the confusing jumping back and forth between Scheme and C
that characterizes our current startup process.
For your edification, here's a fragment of a test program that's in
the test suite for a new version of the QIF importer I'm working on
(this version also supports QIF export, BTW). It uses the non-GUI
'qif-io-core' module and the engine module to import and save as XML a
QIF file without any gui at all:
(define (do-file filename)
(use-modules (gnucash gnc-module))
(gnc:module-load "gnucash/engine" 0)
(gnc:module-load "gnucash/import-export/qif-io-core" 0)
(let ((qiffile (qif-io:make-empty-file))
(acct-table (qif-io:make-empty-acct-table)))
;; read the file and look at data formats. we need to do this
;; immediately when loading a file.
(qif-io:read-file qiffile filename #f)
;; this will throw out an exception if there are no possible correct
;; interpretations. we'll correct the ambiguities.
(catch 'qif-io:ambiguous-data-format
(lambda ()
(qif-io:setup-data-formats qiffile))
(lambda (key field-type field-name possible-formats continue-proc)
(simple-format #t "field format: n='~S' t='~S' v='~S' u='~S'\n"
field-name field-type possible-formats
(car possible-formats))
(continue-proc (car possible-formats))))
;; now we need to figure out what information is missing from this
;; file.
(if (qif-io:file-xtns-need-acct? qiffile)
(qif-io:file-set-default-src-acct! qiffile filename))
;; default currency is USD. fix this in GUI.
(let ((commodity
(gnc:commodity-table-lookup
(gnc:engine-commodities) "ISO4217" "USD")))
;; import the bank transactions
(for-each
(lambda (xtn)
(qif-io:bank-xtn-import xtn qiffile acct-table commodity))
(qif-io:file-bank-xtns qiffile))
;; and the investment transactions
(for-each
(lambda (xtn)
(qif-io:invst-xtn-import xtn qiffile acct-table commodity))
(qif-io:file-invst-xtns qiffile))
;; build a gnucash account group
(let ((group (qif-io:acct-table-make-gnc-group
acct-table qiffile commodity)))
;; write the file
(let ((book (gnc:book-new))
(name (simple-format #f "file:~A.gnc" filename)))
(simple-format #t "using book name='~A'\n" name)
(gnc:book-set-group book group)
(gnc:book-begin book name #t #t)
(gnc:book-save book)
(gnc:book-end book)))))
0)