Module system and source tree reorg

[Bill Gribble]

I have been working on a module/plugin system for gnucash.  It's
pretty much finished; I've posted a couple of messages here about the
way it works, and I'll start another thread for discussion of its
design and how it might need to change before a first release.  In
general, it's a wrapper around 'dlopen' with explicit support for
versioning, load/unload/init hooks, and a path-scanning approach to
building a database of available modules.

What I want to address in this thread is how the gnucash source tree 
can be restructured to take advantage of the module system.  I have
an approach I've been following in my source tree, but I'm not 
convinced it's the best.

Here are the general principles I'm trying to go for in my
reorganization and a stab at a source tree structure.  Please 
let me know what you think.

Thanks,
b.g.

---------------------------------------------------------------------

General principles:

1. Modules are the basic organizing unit for the code.  All the code
needed for a module goes in one place, separate from the code for
other modules.  In my source tree, this is in subdirectories of
src/modules/ but eventually it's my goal that *all* of gnucash should
be in modules, so maybe we should just put modules in top-level
subdirs of src/

2. A module should be self-contained.  All the stuff needed to build
the module, including C code, Scheme code, g-wrap specs, g-wrap helper
files, tests, and documentation should all go in one place.  For example,
the "engine" module looks like this:

  src/modules/engine
    Makefile.am
    gw-engine-spec.scm    - g-wrap spec file for just the engine API
    engine-helpers.{c,h}  - g-wrap helper functions for engine types, 
                            from gnc-helpers.{c,h}
    gncmod-engine.c       - the actual module definition stub file 
    gnucash/              - subdir for Scheme module files 
      engine.scm          - (gnucash engine) Scheme module 
      engine-init.scm     - files from old src/scm (loaded by engine.scm)
      gnc-numeric.scm
      commodity-table.scm
      iso-4217-currencies.scm
    engine/               - subdir for 'libgncengine.la' source 
      test/               - tests for libgncengine.la
    test/                 - tests for the "engine" module (libgncmod-engine.la)

What's nice about this is that for the first time all the code needed
to use the engine in an app is in one place, and I can load it up from
a Scheme script like so:

   (use-modules (gnucash gnc-module))
   (gnc:module-system-init)

   (gnc:module-load "gnucash/engine" 0)

   (let ((acc (gnc:malloc-account)))
     ...)
    
Everything else can depend on the engine module by putting a similar
module-load in their own module init functions. Right now, in my
testing of the new QIF importer, I have a 20-line Scheme program that
can read a QIF file and save it as a Gnucash XML file from the command
line.  I would love to be able to really script all the gnucash
components that way: run reports from the command line, run a headless
Gnucash server, etc.

3. Modules should be separately compilable and distributable.  ATM my
source tree depends on a top-level configure script, but I think that
should possibly change (at least for non-core modules). It would be
really nice to get a bunch of contributed "plugins" a la Gimp.  Right
now it would be a little awkward to write a C-only plugin that did
anything useful without linking against the engine library, but it
could be done, and it's easy to write a Scheme-only plugin that does
useful stuff without any gnucash libraries available at compile time.
In the future we will probably want to have a gnucash-devel package
that allows third parties to write and build C plugins outside the
gnucash source tree.

---------------------------------------------------------------------

And here's the start of a proposal for a set of modules and where they
would live in the source tree.  I'd like people to look at it, add to
it, restructure it, and so on.  The idea is that "gnucash" will just
be a little Scheme script that does a lot of "gnc:module-load"s and
starts the gui. 

I'm a little unclear about how to use the naming conventions; at the
moment, each module registers itself with an arbitrary string for a
name, and that's what clients ask for with module-load.  I've been
using "/" as a path-component separator in names with the idea that a
GUI would display available modules in a tree using these path
components, but I sort of like a flat structure in the actual source
tree.

I have not yet addressed the structure of most of the GUI components.
I don't think they should all go together; for example, having a
"gnome-gui-splashscreen" that just depends on a very minimal
"gnome-gui-core" would be nice because it would allow us to load just
enough to get a splash screen up before doing much of anything else.

Also we want to break out the dependencies a little; for example, the
QIF i/o gui probably shouldn't be part of an omnibus GUI module,
because the QIF gui will depend on the non-gui parts of QIF i/o, and
it's silly for the basic GUI to depend on QIF i/o being loaded.  That
would argue for a separate gnome-gui-qif-io module.  Opinions?

The following is a list of what I think should be subdirs of "src".
The format is

subdir-name      gnc-module-name
  description 

gnc-module       n/a               
  the gnc-module library source.  it's not a module itself.  You 
  can either link with libgnc-module.la and call gnc_module_system_init
  from C, or (use-modules (gnucash gnc-module)) (gnc:module-system-init)
  from Scheme, which dynamic-links the library and calls the init function.

gnc-app          n/a
  this would be where the actual gnucash application and utilities
  live.  It may be useful to have a module called 'gnc-util' or something
  for non-gui utility code. 

gnc-component    gnucash/component
  the gnc-component events-and-callbacks interface 

gnc-engine       gnucash/engine    
  the engine itself, with no backends.  Where backends are now directly 
  'dlopen'-ed, dlopen calls would be replaced with gnc_module_loads; 
  'dlsym' can be replaced with gnc_module_lookup

backend-file     gnucash/engine/backend/file     
backend-postgres gnucash/engine/backend/postgres
backend-rpc      gnucash/engine/backend/rpc
  the backends.  A kernel-config-like thing at the top level, or just more
  args to top-level configure, can switch compilation on and off. 

report-core      gnucash/report/core
  non-gui code for representing and running reports.  This would be 
  mostly report.scm plus html-*.scm.  Should be useful on its own
  without GUI.

report-guppi     gnucash/report/guppi
  support for guppi graphs in reports 

reports-text    gnucash/reports/basic
  most of the non-graphing reports we have now

reports-guppi    gucash/reports/guppi
  reports that use guppi 

qif-io-core      gnucash/qif-io/core
  non-gui code for importing and exporting QIF files.  Should be useful 
  without GUI.

register-core    gnucash/register/core
  the toolkit-independent parts of the register code 

register-gnome  gnucash/register/gnome
  the gnome-dependent parts