module system (Re: Sync with gnome releases)

[Bill Gribble]

On Fri, Jul 20, 2001 at 01:50:20PM -0400, Derek Atkins wrote:
> A decent module system would DEFINITELY be a good thing.  Although
> a question remains:  how would the source tree be broken up to support
> building different modules?  Would someone have to rebuild the whole
> tree just to add another module, or could we separate modules out
> into their own packages (source and binary) and build against
> some gnucash-development headers/libraries?

Somewhere in between.  I'm in the process of breaking up the gnucash
source tree now, and cleaning up semantic booboos as I find them, but
I have a pretty good idea of how it could work.  The module system
itself is very straightforward (some might say too simple) but I think
it's good enough.

The basic idea is that a Gnucash module is a lt_dlopen-able shared
library.  In order for the module system to recognize it as a module,
it has to define certain symbols, mainly (1) the module system
interface version it's using (2) version information for itself and
(3) initialization and (optionally) cleanup functions.

My source tree currently has 2 modules: 'engine' and 'qif-io-core'.
qif-io-core is the non-gui part of the QIF importer, which I'm
rewriting.  engine is what you'd expect. 

src/modules/engine contains a g-wrap spec file (covering just the
engine API), a C file of g-wrap helper functions, and the module
definition C file that defines the symbols listed above.  There's a
subdirectory for the engine source code itself, which is untouched.

When I build the engine module, it builds a shared lib
(libgncmod-engine.so) containing the module definition/init code, the
g-wrap stubs, and the g-wrap translators, which is linked against the
engine shared lib (libgncengine.so).

To use it in a C program, I just do this: 

   #include <gnc-module.h>
   ...
   gnc_module_load("gnucash/engine", 0); 

The int arg specifies the required "interface number" of the module.
Each module specifies its version using libtool conventions
(interface:revision:age), so you have a prayer of getting a compatible
module loaded.

Of course, since the symbols are invisible inside the dlopened lib,
this doesn't really help that much unless you link the executable
against the engine itself, or else go through and lt_dlsym the symbols
by hand (as the engine itself does with the various backends).  I
haven't completely figured out how to address this.  For the time
being, if a module depends on another module and you don't want to
peek the dlsyms out by hand, you have to link against the other module
at lib-link time.  This is probably the approach I'll use to convert
the rest of the "core" gnucash system to modules.  

>From Scheme, it works a little better:

   (gnc:module-load "gnucash/engine" 0)
   
   (gnc:do-something-with-engine ...)

Since the engine-module initialization code calls the g-wrap module
init functions (which publish the Scheme bindings for the C symbols),
all the symbols that were g-wrapped are visible on the Scheme side
after loading the engine module.  So as of right now you can write a
Scheme program that just loads the engine and nothing else and does
whatever with it, including open files, crawl thru transactions, etc.

People have asked about Scheme-only modules.  They aren't supported
right now, mainly because I haven't quite figured out how to handle
the versioning and such correctly (i.e., in such a way that the system
can know what the name, version, and description of a module are
without having to completely load it, which can have irreversable side
effects in Scheme).  However, that doesn't mean that you can't have
modules that aren't 99%+ Scheme code.  The qif-io-core module, for
example, is really Scheme-only; all the QIF related code is Scheme.
It requires one stub C file that's all of 1035 bytes long, including
comments.  I don't think that's a terrible burden.

I want to get a little farther before putting this stuff into CVS,
just to make sure I haven't gone completely down the wrong path, but I
think I'll be there before too long.

I'm including my current README just for grins. 

b.g.

This is gnc-module, a plugin/module system for gnucash and related
software.


What is a Gnucash module? 
-------------------------

A gnucash module is a dynamically loadable libtool library that
defines the following symbols:

    /* what version of the module system interface is assumed */
    int  gnc_module_system_interface;

    /* information about the module's version */ 
    int  gnc_module_current;
    int  gnc_module_revision;
    int  gnc_module_age; 

    /* function called to initialize the plugin */
    int  gnc_module_init(void)
    void gnc_module_finish(void)  
   
    /* descriptive information */ 
    char  * gnc_module_path(void);
    char  * gnc_module_description(void); 
    
gnc_module_system_interface is the revision number of the interface
listed above (i.e. the names and type signatures of the symbols each
module must define).  The current revision number is 0; all modules
should assign gnc_module_system_interface = 0.  As the interface
evolves, this should allow us to continue to load older modules.

The current, revision, age triplet describe the module version
according to libtool(1) conventions.  To quote from the libtool info
pages:

  1. Start with version information of `0:0:0' for each libtool library.

  2. Update the version information only immediately before a public
     release of your software.  More frequent updates are unnecessary,
     and only guarantee that the current interface number gets larger
     faster.

  3. If the library source code has changed at all since the last
     update, then increment REVISION (`C:R:A' becomes `C:r+1:A').

  4. If any interfaces have been added, removed, or changed since the
     last update, increment CURRENT, and set REVISION to 0.

  5. If any interfaces have been added since the last public release,
     then increment AGE.

  6. If any interfaces have been removed since the last public release,
     then set AGE to 0.

gnc_module_path should return a newly-allocated string containing the
"module path". The module path is a logical path with elements
separated by /.  The path will be used to structure views of available
modules so that similar modules can be grouped together; it may or may
not actually correspond to a filesystem path.  The last element of the
module path is the name of this module.  For example,
  char * path = "gnucash/core/engine";
defines the "engine" module, which is in the gnucash/core group of 
modules. 

gnc_module_description should return a newly-allocated 1-line
description of what the module does.  This can be displayed by GUI
elements to allow users to select modules to load.


Initializing the module system
------------------------------

Somewhere at program startup time, you need to call
gnc_module_system_init from C.  This does two things:

  - publishes the g-wrapped module system API to the Scheme side
  - scans the GNC_MODULE_PATH for libtool libraries and builds a database 
    of available modules 

You can also call gnc:module-system-init from Scheme; this is not a 
g-wrapped function.  It uses Guile's dynamic-link to open the 
libgncmodule.la library and call gnc_module_system_init.  

You can rebuild the module database at any time (say, if you know a
new module has been installed or the user has changed the module path
via some in-program mechanism) by calling gnc_module_system_refresh.

Loading modules
---------------

>From C call gnc_module_load(path, interface), or gnc:module-load from
Scheme.  This returns a GNCModule (<gnc:module>) if a qualifying
module was successfully loaded, #f / FALSE otherside.  GNCModule is an
opaque type.

A qualifying module is any module whose gnc_module_path matches the
path specification and for whom "interface" falls between
gnc_module_current and (gnc_module_current - gnc_module_age).  If
multiple modules qualify, libtool's rules are used to determine which
to load: the larger of gnc_module_interface, gnc_module_age, and
gnc_module_revision take precedence, in that order.

Module initialization
---------------------

The first time a module is loaded, its gnc_module_init function is
called.  Any startup/initialization code should be defined in this
function.  If this module depends on other modules, put the necessary
gnc_module_load calls in the init function.

If gnc_module_init returns FALSE, the module is not loaded.  Any
cleanup must be done within gnc_module_init before returning.