[GNC] File Format Documentation (Bug 777893)

Adrien Monteleone adrien.monteleone at lusfiber.net
Wed Aug 15 13:15:56 EDT 2018


Additional sections?

What you have here is the overview.

So perhaps sections which cover those details are in order for further reading. The overview though should allow someone to make a quick decision at least to determine if they want to investigate further or not without bogging them down in terminology or issues that might not pertain to them.

I’m not sure if the better organization would be sections for each of the 2 main (and then 3 sub) backends or by issue/comparison point. The question there might be, “What is the next logical question after reading the overview?” But of course, this answer will vary. (which is why hypertext is maybe better for this than a static typeset document, but the info does need to be included somehow)

As I noted on the table suggestion, maybe the docs simply reference the relevant wiki pages for more reading, as they can be more easily updated and can easily cross link to other info.

Regards,
Adrien

> On Aug 15, 2018, at 11:42 AM, David T. <sunfish62 at yahoo.com> wrote:
> 
> Adrien,
> 
> As I look at implementing your suggestion of a brief comment paired with the table, I encounter a problem. Specifically, many elements of comparison are entirely unexplained anywhere else (indeed, this was the inspiration to take this on in the first place). It seems to me that some kind of discursive explanation is necessary for many of the points. For example, simply noting in a table that additional software is needed for MySQL, for example, leaves the question of what software to install open—never mind discussing the additional responsibilities that managing a MySQL instance incurs. I am not sure how to include this, if not in the text of the Guide.
> 
> While I see your point about summarizing this information, I must ask how (and where) we should address these other issues? 
> 
> David
> 
>> On Aug 15, 2018, at 8:59 AM, David T. via gnucash-user <gnucash-user at gnucash.org> wrote:
>> 
>> Adrrien,
>> 
>> Thanks for the additional input.
>> 
>>> On Aug 14, 2018, at 11:00 PM, Adrien Monteleone <adrien.monteleone at lusfiber.net> wrote:
>>> 
>>> Are the dirty details really necessary?
>>> 
>>> I don’t see that they help the user (who may not be well versed in the issues) to make a decision. I would think simpler in this case would be better. If it is clear that GnuCash does not yet take full advantage of SQL and that specifically, you can’t directly edit the db or have multi-user, I think that gets the point across. Normalization, or the app not adhering to an MVC type pattern aren’t going to be issues I think most people looking at this explanation are going to be immediately concerned about. (and many might not even know what they mean) Stephen’s suggestion seems much less confusing overall.
>> 
>> I will leave the technical details for the wiki. Currently, there is a (rather dated) page delineating the XML format; I will create a parallel page for the SQL format(s) that can collect many of these pieces of information.
>> 
>> 
>>> 
>>> Really, a simple overview paired with the comparison table would do the trick.
>>> 
>>> Additionally, the comparison table might serve better with each SQL backend represented, since that’s the next question for someone contemplating that route. And in that case, sqlite does not necessarily require additional software, or expertise with a DBMS. (it’s included in the default distribution of the software on Mac and Linux, you just have to choose it on save, and you don’t really need to know how it works, unlike the other two.) Finally, perhaps presenting the table as a list of features/factors with check marks or Xs, could be easier to scan and digest.
>> 
>> I’ll look into this modification as well.
>> 
>>> 
>>> Regards,
>>> Adrien
>>> 
>>>> On Aug 15, 2018, at 12:09 AM, David T. via gnucash-user <gnucash-user at gnucash.org> wrote:
>>>> 
>>>> Stephen,
>>>> 
>>>> 
>>>>> On Aug 14, 2018, at 4:50 PM, Stephen M. Butler <kg7je at arrl.net> wrote:
>>>>> 
>>>>> On 08/14/2018 04:24 PM, David T. wrote:
>>>>>> Steve,
>>>>>> 
>>>>>> Thanks for your input. I believe we’re in agreement here; I wasn’t trying to suggest that GC become a DBMS, but rather it would learn to utilize DBMS features that many seem to expect.
>>>>>> 
>>>>>> I’m not great with the appropriate terminology to use. Do you have language I could use that would more accurately reflect the direction that GC is headed?
>>>>>> 
>>>>>> David
>>>>> 
>>>>> I hoped as much.  Let's take this pair of sentences:
>>>>> 
>>>>> Use of SQL formats for storage implies to many that GnuCash is database management software (DBMS). While that is a long term goal of the development team, GnuCash as currently designed is not DBMS.
>>>>> 
>>>>> And change them to this:
>>>>> 
>>>>> Use of a SQL back-end engine for storage implies to many that GnuCash
>>>>> has fully implemented DBMS features including multi-user and incremental
>>>>> data manipulation.  That is a long-term goal of the development team. 
>>>>> However GnuCash does not currently implement these features.
>>>>> 
>>>> 
>>>> This is great; I will use it.
>>>> 
>>>>> Then I think this next section can disappear entirely:
>>>>> 
>>>>> Many features that users of DBMS expect from a DBMS are not implemented in GnuCash. The GnuCash data schema is not normalized. Significant elements of the data logic is implemented in code, rather than data structure.
>>>>> Gnucash uses the SQL back end to load the entire data store into memory in the same manner as the XML back end. Consequently, use of the SQL back end does not enable simultaneous multi-user access to a GnuCash file, although that is one long term goal.
>>>>> 
>>>> 
>>>> I’m not sure I want to remove these comments, as they are issues of significance that have come up on the lists over the years. How about this:
>>>> 
>>>> The GnuCash data schema is not normalized. Moreover, significant elements of the data logic is implemented in code, rather than data structure. Finally, Gnucash uses the SQL back end to load the entire data store into memory in the same manner as the XML back end. Consequently, use of the SQL back end does not enable simultaneous multi-user access to a GnuCash file.
>>>> 
>>>>> 
>>>>> And I will change this piece:
>>>>> 
>>>>> One benefit of the SQL back end is that it saves changes incrementally, and every change is committed to the back end as it happens. This is in contrast to the XML back end, which only writes to the data file when the user invokes the Save command.
>>>>> 
>>>>> To say:
>>>>> One benefit of the SQL back end is that it saves changes to the back end as they happen. This is in contrast to the XML back end, which only writes to the data file when the user invokes the Save command. 
>>>> 
>>>> How about:
>>>> One benefit of the SQL back end is that it commits changes to the file immediately. This is in contrast to the XML back end, which only writes to the data file when the user invokes the Save command.
>>>> 
>>>> The final result for this section is this:
>>>> ———————  
>>>> Use of a SQL back-end engine for storage implies to many that GnuCash has fully implemented DBMS features including multi-user and incremental data manipulation.  That is a long-term goal of the development team. However GnuCash does not currently implement these features.
>>>> The GnuCash data schema is not normalized. Moreover, significant elements of the data logic is implemented in code, rather than data structure. Finally, Gnucash uses the SQL back end to load the entire data store into memory in the same manner as the XML back end. Consequently, use of the SQL back end does not enable simultaneous multi-user access to a GnuCash file.
>>>> One benefit of the SQL back end is that it commits changes to the file immediately. This is in contrast to the XML back end, which only writes to the data file when the user invokes the Save command.
>>>> ———————
>>>> 
>>>> Better?
>>>> David
>>>> 
>>>>> 
>>>>> 
>>>>> Hope this helps.
>>>>> 
>>>>> --Steve
>>>>> 
>>>>>> 
>>>>>>> On Aug 14, 2018, at 3:50 PM, Stephen M. Butler <kg7je at arrl.net> wrote:
>>>>>>> 
>>>>>>> On 08/14/2018 03:05 PM, David T. via gnucash-user wrote:
>>>>>>>> Hello,
>>>>>>>> 
>>>>>>>> In response to Bug 777893 (https://bugs.gnucash.org/show_bug.cgi?id=777893 <https://bugs.gnucash.org/show_bug.cgi?id=777893>), I have written a more detailed description of the storage choices available to users for insertion into the Tutorial & Concepts Guide at section 2.5. Given the extent of the text, I am including it here so that the broader community can offer suggestions for improvement. Note that I will insert appropriate encoding once I finalize the content.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> David T.
>>>>>>>> 
>>>>>>>> ————————————————————
>>>>>>>> Proposed text for Tutorial & Concepts Guide section 2.5
>>>>>>>> ————————————————————
>>>>>>>> 
>>>>>>>> 2.5 Storing your financial data
>>>>>>>> 2.5.1 Overview
>>>>>>>> GnuCash offers several formats for storing your financial data. The default file storage format is XML, while a number of flavors of SQL storage are available. Users can choose a file format from the File Save and File Save As dialogs.
>>>>>>>> The primary GnuCash storage format is an XML file. The file is by default compressed with gzip, which is a preference that is set at Edit→Preferences→General→Use file compression. 
>>>>>>>> GnuCash also supports SQL storage via the DBI back end. It supports PostgreSQL, MySQL and SQLite3 databases.
>>>>>>>> 
>>>>>>>> Storage Comparison
>>>>>>>> As noted, GnuCash allows storage in either XML or SQL formats. Each of these formats has benefits and shortcomings that the user should consider for their needs and abilities. 
>>>>>>>> The XML format is the most stable and established format, and for this reason, it is the recommended format for most users. SQL storage was added for the 2.4 release and has become an increasingly popular choice for users.
>>>>>>>> Use of SQL formats for storage implies to many that GnuCash is database management software (DBMS). While that is a long term goal of the development team, GnuCash as currently designed is not DBMS. It is a financial application that can store its data in SQL files.
>>>>>>> I would strongly suggest that GnuCash never (triple underscore) become a
>>>>>>> DBMS engine.  I would encourage it to be an application that uses a
>>>>>>> strong DBMS engine (which it already does) as a back-end and that the
>>>>>>> team would apply data normalization and multi-user techniques to the
>>>>>>> front-end application.
>>>>>>> 
>>>>>>> Note the difference between a front-end application and the back-end
>>>>>>> DBMS engine.
>>>>>>> --Steve  (retired Oracle DBA and former application developer)
>>>>>>>> Many features that users of DBMS expect from a DBMS are not implemented in GnuCash. The GnuCash data schema is not normalized. Significant elements of the data logic is implemented in code, rather than data structure.
>>>>>>>> Gnucash uses the SQL back end to load the entire data store into memory in the same manner as the XML back end. Consequently, use of the SQL back end does not enable simultaneous multi-user access to a GnuCash file, although that is one long term goal.
>>>>>>>> One benefit of the SQL back end is that it saves changes incrementally, and every change is committed to the back end as it happens. This is in contrast to the XML back end, which only writes to the data file when the user invokes the Save command. 
>>>>>>>> The SQL back end does allow users with SQL experience to write queries against the data to create custom reports without using Gnucash’s internal report system. It is important to note that modification of GnuCash data using external access modes is strongly discouraged by the development team, and damages to a GnuCash data file that result from such modifications are strictly the responsibility of the user.
>>>>>>>> 
>>>>>>>> Storage Comparison Table
>>>>>>>> 
>>>>>>>> XML     						SQL
>>>>>>>> Default						Optional
>>>>>>>> Requires no additional software	Requires additional software
>>>>>>>> Requires no additional expertise	Requires expertise with DBMS 
>>>>>>>> Compressed  					Uncompressed
>>>>>>>> Save on command				Save on commit
>>>>>>>> Uses log files					Does not use log files
>>>>>>>> Not multi-user					Not multi-user
>>>>>>>> Limited external processing		External processing through SQL queries
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>>> -- 
>>>>> Stephen M Butler, PMP, PSM
>>>>> Stephen.M.Butler51 at gmail.com
>>>>> kg7je at arrl.net
>>>>> 253-350-0166
>>>>> -------------------------------------------
>>>>> GnuPG Fingerprint:  8A25 9726 D439 758D D846 E5D4 282A 5477 0385 81D8
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> gnucash-user mailing list
>>>> gnucash-user at gnucash.org
>>>> To update your subscription preferences or to unsubscribe:
>>>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>>>> If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>>>> -----
>>>> Please remember to CC this list on all your replies.
>>>> You can do this by using Reply-To-List or Reply-All.
>>> 
>>> 
>>> _______________________________________________
>>> gnucash-user mailing list
>>> gnucash-user at gnucash.org
>>> To update your subscription preferences or to unsubscribe:
>>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>>> If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>>> -----
>>> Please remember to CC this list on all your replies.
>>> You can do this by using Reply-To-List or Reply-All.
>> 
>> _______________________________________________
>> gnucash-user mailing list
>> gnucash-user at gnucash.org
>> To update your subscription preferences or to unsubscribe:
>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>> If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>> -----
>> Please remember to CC this list on all your replies.
>> You can do this by using Reply-To-List or Reply-All.
> 
> 




More information about the gnucash-user mailing list