GnuCash XML spec (for non-C languages?)

[John Ralls]

On Nov 5, 2012, at 1:55 PM, Geert Janssens <janssens-geert at telenet.be> wrote:

> On 04-11-12 23:05, Christian Stimming wrote:
>> Am Donnerstag, 1. November 2012, 09:43:30 schrieb John Ralls:
>>> On Nov 1, 2012, at 9:32 AM, Geert Janssens <janssens-geert at telenet.be>
>> wrote:
>>>> On 01-11-12 17:13, Derek Atkins wrote:
>>>>>> Also, what is the policy of GnuCash towards manipulating the XML.
>>>>>> 
>>>>>> Is modifying the XML also actively discouraged?
>>>>> Yes.  All data 'writes' should only happen through the GnuCash API.
>>>> Which means there are currently 3 supported ways to modify the data:
>>>> - using the GnuCash application's gui
>>>> - you can write a guile/scheme script that uses the GnuCash API
>>>> - if python bindings are enabled on your platform you can also write a
>>>> python script that uses the GnuCash API.
>>> There's a fourth: You can write a program in any language which can load a C
>>> library.
>> We used to be somewhat proud of this statement - being able to offer a C
>> library that can handle the data store correctly. However, I think the times
>> have changed. By they way, the gnucash API isn't only a C library, but
>> instead, it's a C library requiring glib - any new platform would first have a
>> glib being ported to it.
>> 
>> Today, we have all those new mobile devices. They might even allow C
>> libraries, but if they do, it would be a C library for specific architectures
>> with significantly smaller target audience than the whole mobile platform in
>> mind. Or in other words, one would have to cross-compile a whole set of
>> different C libraries to cover all the mobile devices using a particular
>> platform. And for the gnucash API this all wouldn't even work as long as there
>> isn't a glib on that platform!
>> 
>> Because of this new situation today, I think it would be quite useful to be
>> able to modify our original point of view concerning access to the datastore.
>> I think it would be quite useful to find some sort of datastore access layer
>> that is *not* forced to be a C library. In fact, it shouldn't be tied to any
>> specific programming language. If it were possible to postulate the structure
>> of the datastore in such a language-independent way, it would enable other
>> languages and platforms without C libraries to offer gnucash data display and
>> manipulation. Such as Ngewi's Android app, directly accessing a SQL database
>> with gnucash data. Or any app on any of those other well-known mobile
>> platforms. They all have in common that it is impossible to use the gnucash C
>> library. They all have in common that it's a huge improvement for users to be
>> able to do their bookkeeping also from those devices.
>> 
>> So IMHO the logical next step is to find some new formulation of the gnucash
>> datastore where the data manipulation is no longer solely tied to a C library.
>> I know this step isn't particularly easy, but I think the time has come to no
>> longer outright refuse such a step.
>> 
>> Regards,
>> 
>> Christian
>> 
> I agree with your point of view that there is a whole new breed of devices flooding the market for which GnuCash as we know it is not really well suited, both from a technology point of view (how portable is our code ?) or in flexibility to adapt to new form factors.
> 
> I'm a curious to learn what our options are to realistically improve this. Below is a lot of thinking out loud about the situation we are in now.
> 
> Reading the above, my first urge tends to be "rewrite the whole thing from scratch, except cleaner and more perfect in a platform independent way". Which I immediately reclassify as not very realistic considering the limited time the current team of volunteers has to spend on the project.
> 
> We could also try to "clean up the current Augean stable" as John nicely puts it. I thought this was more or less what we decided to do the previous time we did some introspection in the state of the gnucash code base. An important first step was to write a good set of unit tests covering all critical engine code. From there on qof could gradually be replaced with the more standard gobject framework. The unit tests should prevent most of the regressions. But I'm under the impression the unit test work has stalled mostly. And if at some point in time Android will be considered an important primary platform to run GnuCash on, we could wonder if this work is still worthwhile ? From John's mail I understand that ios and WindowsRT would not be an issue if we continue to work in C.
> 
> An important factor to help evaluate this in my opinion is to look really close at the way the new form factors work. GnuCash has always been and still is a desktop application, and firmly relies on that desktop paradigm. I mean, it is a tool meant to be used by one user at the time and intended for local data stores. Yes you can use a remote database as a data dump. There's no intelligence in the database layer at all though. And yes, you can access a datafile on a network share. But those are all part of a desktop experience. Tablets (and smartphones) on the other hand are heavily vested in a cloud metaphor. Like it or not. Very few apps will let you choose where on the tablet you want to store your data. Most of them are backed by a web service which conveniently "take all the complexity of file management out of your hands".  People used to the desktop metaphor tend to find that ridiculous, but my observation of less computer literate people around me confirm this works. Such people really like the idea of just tapping an app, passing a one-time setup consisting of entering an e-mail address and choosing a password and go on happily using the app. They are not interested where their data is stored and that it's most likely not on the tablet itself.
> 
> So far the general idea. Where I want to get at is what Colin suggested: if you want to do something in the tablet world, you have to think the tablet way, which means "cloud". For clarity, when I say "cloud" I mean your tablet app itself should be light-weight and be backed by a powerful online service. This service can run either hosted on some public cloud provider, or on your own server at your office. That's irrelevant here. So I'm not against the idea of a web frontend for GnuCash.
> 
> So that is one way to see GnuCash on tablets: a web service with an associated client app on the tablets. This would allow us to reuse much of the existing code base (the engine). Ideally, that would be split into a separate library with a well defined api. This api could then equally be used for the current GnuCash desktop application, a web frontend, perhaps even a full featured GnuCash for tablets application on tablets that run C natively, and so on. As far as I'm concerned I would even love to see our collegues (like KMyMoney, Skrooge,...) use our own library to import/export from/to gnucash. The same api could also be exposed via scheme and python (which are there now already, but can then be cleared up as well), ...
> 
> As for the work involved. It is still huge. I wouldn't know what is required to create a web interface on top of GnuCash. But there are at least some challenges I can think of:
> * basic web protocols such as http are stateless, how does the server know which client wants to access which data file ? Cookies ? Perhaps.
> * Web clients generally don't inform the server they have finished. Each server interaction is more or less atomic on it's own. So how will the server know a data file can be closed, has to be saved, ... All desktop metaphors that are clumsy in a web interface world. Xml is useless as a backend in this context.
> * The database backends go partly in the right direction, but still aren't adapted to be used as web server backends. The current sql backends still load the whole dataset in the beginning because the GnuCash engine was never written with a database as backend in mind. This would have to be fixed.
> * Simultaneous multi-user access is definitely a basic requirement. Even if there's only one real user, he/she will most certainly want to access the financial data from both the tablet and the desktop without having to remember to log out on the one first before logging in on the other.
> And so on...
> 
> Anyway, that's the option I see currently. I'd love to hear other ideas though.

OK.

Engine unit testing has gotten set aside this summer by my realization that we're running into the 2038 date issue rather sooner than we expected, because it's breaking long-term SXes. Dealing with that turns out to be a lot of work, but I hope to wrap it up by the end of the year and get back to testing Engine and QOF.

That aside, there are two ways to approach cloud storage: A web service, where all of the heavy-duty thinking happens on a server somewhere, is one, but just storing on a server somewhere is another. Both Apple and Microsoft include direct support for this in their latest offerings, but we'd need to write QofBackends to make use of it. There's probably a way to get there from Android as well, but they don't make it easy. 

Having a true web-app would indeed finesse that problem. The normal way to implement that is AJAX, and it can be state-ful. The state exists on the server side, where a database transaction is begun and either committed or rolled back, perhaps after a time out. All standard stuff. Requires getting to a true transactional database program, though. Simultaneous users comes for free with that. That's my long-term plan… 

Regards,
John Ralls