Difference between revisions of "Development Process"

From GnuCash
Jump to: navigation, search
(→‎Changeset Auditing Process: Change backport id)
(Update branches, build system (docs), …)
 
(33 intermediate revisions by 7 users not shown)
Line 1: Line 1:
−
This page describes the development process for developing [[GnuCash]] code.  It talks about how we use [[Subversion]] and in particular how we use branches to manage the [[GnuCash]] source code and release process.
+
This page describes the development process for developing [[GnuCash]] code.  It talks about how we use [[Git]] and in particular how we use branches to manage the [[GnuCash]] source code and release process.
  
 
==Development Process Goals==
 
==Development Process Goals==
−
 
 
The goal of this process is to limit the number of potential bugs that enter the stable release branch(es) due to changes made to the sources.  We want to make sure that changes made to releases have been audited and vetted BEFORE they are commited to the release branches.  At the same time we want to try to keep the main development branch as stable as practical.
 
The goal of this process is to limit the number of potential bugs that enter the stable release branch(es) due to changes made to the sources.  We want to make sure that changes made to releases have been audited and vetted BEFORE they are commited to the release branches.  At the same time we want to try to keep the main development branch as stable as practical.
  
 
In particular, we want to prevent "bad releases".  Those are releases where a bugfix inadvertantly breaks other code.
 
In particular, we want to prevent "bad releases".  Those are releases where a bugfix inadvertantly breaks other code.
 +
 +
==Documentation==
 +
Documentation should ideally start before coding ...
 +
 +
There are 3 target groups for documentation:
 +
;Code: For current and future developers add comments in the source files.
 +
;API: see [[Doxygen]].
 +
;User: has again 2 sources.
 +
:; In Code: add good tooltips
 +
:; In the Docs:
 +
New or reworked features should also be documented as it turns out, some bugs are a problem of communication between user and developer. If you do not have the resources to [[Documentation_Update_Instructions|update the documentation]] yourself, you should at least assign a [[Bugzilla]] entry to the documentation team with the essential information.
  
 
==Branches==
 
==Branches==
 +
Branches in [[Git]] are cheap, therefore we use branches to our advantage. There are almost always two "interesting" branches in use:
 +
* ''stable'' — the current stable branch leading to then next ''minor'' release,
 +
* ''future'' — the development branch leading to then next ''major'' release, but only after it differs from stable.
 +
 +
When close to a major release there will be a third "interesting" branch:
 +
* 'unstable' — the preview branch for the upcoming major release
  
−
Branches in [[Subversion]] are relatively cheap, therefore we should use branches to our advantage. There are always two "interesting" branches in use:
+
Older release series are labelled with the release number, e.g. ''2.4''. Those branches are normally closed to new work shortly after a new ''stable'' branch is created from ''unstable'' at the beginning of a new release series.
  
−
* ''trunk'' -- the main development branch
+
Developers may also have other branches in use for development or bugfixing.  This is described later in this document. These will generally '''not''' be pushed to the main repository.
−
* ''branches/<release>'' -- the current release branch(es)
 
  
−
Developers may also have other branches in use for development or bugfixing. This is described later in this document.
+
==Dependencies==
 +
Changing dependencies should usually happen on the future branch.
 +
;Places to change (minimal) versions:
 +
: Buildsytem
 +
::Code: [https://github.com/Gnucash/gnucash/blob/stable/CMakeLists.txt CMakeLists.txt] or
 +
::Docs: [https://github.com/Gnucash/gnucash-docs/blob/stable/CMakeLists.txt CMakeLists.txt]
 +
: README
 +
::Code: [https://github.com/Gnucash/gnucash/blob/stable/README.dependencies README.dependencies] and
 +
::: it's wiki derivate [[Dependencies]],
 +
::Docs: [https://github.com/Gnucash/gnucash-docs/blob/stable/README README]
 +
: Wiki Templates
 +
::Some version numbers are defined in [https://wiki.gnucash.org/wiki/index.php?title=Special%3APrefixIndex&prefix=&namespace=10 Templates] for easier updates of links etc.
  
 
==Developing New Features==
 
==Developing New Features==
 +
All new development work should target the ''future'' branch.  This includes new features and enhancements.  Small changes can be commited directly to ''future''.  Larger features or architectural changes should happen on a feature-branch; once that feature is working it can get merged back into ''future''.
  
−
All development should target the ''trunk'' branch. This includes new features, bug fixes, and enhancements.  Small changes can be commited directly to ''trunk''.  Larger features or architectural changes should happen on a feature-branch; once that feature is working it can get merged back into ''trunk''.
+
Bug fixes should be built against the ''stable'' branch. See [[#Fixing Bugs|Fixing Bugs]] below for details.
  
−
At any particular time, ''trunk'' should be "as stable as possible".  We use feature branches to limit the chance that a feature never completes.  If development of a feature were done solely in ''trunk'' and that feature never got completed it could leave ''trunk'' in a state where it doesn't compile, or it fails the regression tests.
+
At any particular time, ''future'' should be "as stable as possible".  We use feature branches to limit the risk that a feature never completes.  If development of a feature were done solely in ''future'' and that feature never got completed it could leave ''future'' in a state where it doesn't compile, or it fails the regression tests.
  
−
If you are at all questioning whether to develop directly in ''trunk'' or in a branch, do it in a branch.
+
If you are at all questioning whether to develop directly in ''future'' or in a branch, do it in a branch.
 +
 
 +
As a general rule, development branches should ''not'' be pushed back to the main repository. If one needs to publish a development branch, fork the [https://github.com/Gnucash/gnucash Github repository] to your personal account and publish the branch there.
  
 
===Minor Changes===
 
===Minor Changes===
 +
Developers with the requisite privs may push minor changes directly to ''stable'' or to ''future'' as appropriate without a feature branch.
 +
 +
===Major Changes===
 +
Use your judgement about whether your major change should be done directly on ''future/stable'' or whether you should use a feature branch.  We trust you.  Just remember, smaller changesets are easier to audit. When possible try to ensure that each commit can compile; that makes it much easier to bisect when tracking down a regression.
 +
 +
===Backwards-Incompatible Schema Changes===
 +
To make the policy clearer, backwards-incompatible schema changes in an unstable release, e.g. 5, require a change to the previous stable release, e.g. 4, to provide a read facility for the new schema.
 +
 +
Often it is easier to add a new KVP which will be simply ignored by older versions than to change the schema. But then it would become harder to get the [http://en.wikipedia.org/wiki/Database_normalization database normalized] for the SQL backend.
 +
 +
===LibGnuCash API Stability===
 +
<!-- The discussion leading to this section may be found at {{URL:Lists}}logs/2018/08/06.html#T13:04:30 -->
 +
Historically GnuCash has been developed as a stand-alone library with no concern for stability or backwards compatibility of the core API in spite of simultaneously insisting that the only safe way to operate on GnuCash data is to write code using the somewhat published API in C, Scheme, or Python. Over the first 20 years or so API has mostly been added and seldom pruned, so at as we get into converting more interesting components to C++ it's time to clean up and de-duplicate the API.
 +
 +
As far as we're able to tell no-one has ever written a consumer of the GnuCash C API outside of GnuCash itself, so during the conversion we're not going to worry about it. Everything that isn't Gui or Report will be rewritten in C++ and the C++ API will be new, and once it's internally stable we can begin to treat it like a real library with API guarantees. The C API necessary to interact with the Gtk GUI will remain as long as we retain the latter but there will be no stability guarantee for that part.
 +
 +
The bindings are another matter entirely. Much of the existing C API is also exported to Scheme and Python via SWIG. We know of many cases where users have written custom reports and a few cases of users writing Python extensions. Those users deserve reasonable stability in the API, hence the following policy.
  
−
Minor changes may be commited directly to ''trunk''.
+
====Policy====
 +
* Functions must not change signature (except in C++ which supports overloading). If one needs to do the same thing with different types, make a new function with a different name.
 +
* Functions may only be deleted in the ''future'' branch. Functions to be deleted must first be marked deprecated in the ''stable'' branch '''at least 6 months before the major release'''.
 +
* Deprecated functions should be rewritten as wrappers of their replacement functions. If the function is no longer needed in the C library the wrapper should be in the SWIG interface file (e.g. <tt>engine.i</tt>).
  
−
===Major Changes===
+
====Marking a function as deprecated====
 +
* C: <tt>__attribute__((deprecated))</tt> (gcc/clang) or <tt>__declspec(deprecated("explanation/redirection string"))</tt> (Visual Studio)
 +
* C++: <tt><nowiki>[[deprecated("explanation/redirection string")]]</nowiki></tt> (Requires C++14, use the C method in stable)
 +
* Python:  <tt>warnings.warn("explanation/redirection string", DeprecationWarning, stacklevel=2)</tt>
 +
* Scheme: <tt>(issue-deprecation-warning "explanation/redirection string")</tt>
  
−
Use your judgement about whether your major change should be done directly on ''trunk'' or whether you should use a development branch.  We trust you.  Just remember, smaller changesets are easier to audit.  Also keep in mind that '''svn blame''' is going to point to the "merge" changeset and not the individual changeset after your development branch is merged into ''trunk''.
+
'''Python Note:''' <tt>DeprecationWarnings</tt> are disabled by default, so developers of Python programs using GnuCash bindings should test by invoking python with <tt>-Wd</tt> or setting <tt>PYTHONWARNINGS=default</tt> in the shell environment.
  
 
==Fixing Bugs==
 
==Fixing Bugs==
 +
In general bug fixes should always target the oldest active branch that features the bug. If the bug is present in the current stable version of GnuCash, the bug should be fixed on ''stable'' (or a sub branch thereof). If the bug exists only in the development version of GnuCash, the bug should be fixed on ''future'' only.
 +
 +
The exception here would be a bug in the stable branch for which the fix itself is a very complex or large body of code. Since this code has a higher risk of introducing new bugs in itself it should not be applied on the ''stable'' branch. Instead such a bug can only be fixed on the ''future'' branch (unstable version).
 +
 +
Since we frequently merge the ''stable'' branch into ''future'', please do a test merge of ''stable'' into ''future'' in your local repository to ensure that your patch merges cleanly.
 +
 +
If it doesn't and you don't know what to do about it, say so on the bug report when you attach your patch so that a developer can help you asses the situation and decide how to resolve it.
 +
 +
===Bugfix Branches===
 +
Many times a bugfix is simple.  In that case the bug can be fixed directly on the ''stable'' branch and the changeset can be merged forward to the ''future'' branch so future major releases carry it as well.
 +
 +
If the bugfix is sufficiently complicated then that bugfix warrants a bugfix branch.  The developer can create a branch off the release branch (''stable'') to work on the bugfix, and once that's been tested that branch can be merged back into the release branch.
 +
 +
==Release Version Numbering==
 +
GnuCash uses two version number schemes:
 +
* For versions up through 2.7.8, each GnuCash version number consists of 3 numbers separated by a period, in format '''Fundamental.Major.Bugfix''' E.g. 2.4.11
 +
* Beginning with version 3.0, released in April 2018, each version number consists of only 2 integers separated by a period,
 +
meaning '''Major.Minor''', e.g. 3.1.
 +
 +
 +
===Fundamental===
 +
The only fundamental change, from 1 to 2, involved changing most of the code from Scheme to C and upgrading the GUI from Gtk1 to Gtk2. Since we used this so infrequently we decided to switch to a two-number release.
  
−
Contrary to some beliefs, bug fixing is still development; it's just a different type of development. Indeed, whenever a bug exists that bug most likely exists in all branches, so it should get fixed in all branches.  Therefore, the bugfix process should provide a way to fix a bug in ''trunk'' and, when necessary or appropriate, also on release branches.
+
===Major===
 +
Major releases are planned every 2 - 4 years. See [[Release_Schedule|the Release Schedule]] for the development teams forecast.
  
−
Fixing bugs leads to its own set of issues, especially due to the fact that ''trunk'' will diverge from the release branches.  Some may think that it would be easier to just commit bugfixes directly to the release branch and then merge back into ''trunk'', but while this might require fewer command-line operations, it has many pitfalls.
+
There are 2 types of Major releases, Stable and Unstable.
  
−
===Pitfalls of fixing directly on a release branch===
+
====Stable====
 +
:Stable releases are intended for use by users with their live data. They have one or two-digit minor versions, e.g. 3.1 or 3.21.
  
−
* it requires unvetted changes be committed to the release branch
+
====Unstable====
−
* it fails to work well if we have multiple release branches
+
:Unstable release numbers have three digit minor versions beginning with 900, e.g. 3.902.
−
* as ''trunk'' diverges, the merging can become even more challenging. While cherry-picking changesets to pull into a release branch requires typing more commands, merging a large set of changes from the release branch back into ''trunk'' can become a huge tangled mess if any of the changes don't apply cleanly.
 
−
* when looking at a change in ''trunk'' to see why a line of code changed '''svn blame''' would only show that the change came from a "merge from stable branch" instead of showing who actually made the change or why.
 
  
−
It is true that as ''trunk'' diverges some bugfixes might require a lot of work to "backport" to the release branch (assuming that the fix even IS a backport).  In that case, the developer should use a bugfix branch (based off the release branch) to work on the fix. However, that same bugfix if made in the release branch would require the same amount of work to merge into ''trunk'' so it's not saving any work to do it that way, and there are many drawbacks to the approach.
+
:Unstable releases are the precursors to a stable release, and are intended only for testing, by either developers or interested community members who don't have the time or skills to build GnuCash from source. They are meant to expose the developers' work to a wide audience, so changes can be tested in a range of environments, before being included in a stable release.
  
−
===Bugfix Branches===
+
:Unstable releases come late in a major development cycle. By the time unstable releases are issued, relevant longer term modifications should be nearly done (except for some bug fixing).
  
−
Many times a bugfix is simple and the code has not diverged.  In that case the bug can be fixed on ''trunk'' and the changeset can be merged (cherry picked) directly onto the release branch. This will happen more often than not early on in the development cycle because ''trunk'' and the release started the same.  As time goes on it may happen less often depending on the actual change and how much the code has diverged.
+
:Unstable releases are very short-lived, normally only six months or so, with new releases every month and a final pre-stable release two weeks before the stable release. If a significant problem surfaces in that last release, the developers will focus on that problem and do another pre-stable with the stable to follow two weeks later.
  
−
If the code has diverged enough that a simple bugfix can't be applied, or the bugfix is sufficiently complicated, then that bugfix warrants a bugfix branch.  The developer can create a branch off the release branch to work on the bugfix, and once that's been tested that branch can be merged back into the release branch.  Note that the bug should STILL be fixed in ''trunk'' first.
+
:Due to the possibility of data corruption, unstable releases should only be used on a '''copy''' of live GnuCash data.
  
−
===Changeset Auditing Process===
+
===Minor===
 +
Minor releases are made typically every 3 months; see [[Release_Schedule|the Release Schedule]]. Minor releases contain fixes to reported problems and small features that the development team consider low risk for incompatibility.
  
−
All changes to a release branch must be audited.
+
Minor releases should never break file compatibility with other releases in the same major version.
−
* Before any changes may be made to a release branch, the changeset should exist either in ''trunk'' (if it's a simple fix) or a specific bugfix branch (if the changeset is complicated enough that it just can't apply cleanly).  This way the change is already in [[Subversion]] and has a chance to be vetted before pulled onto the release branch.
 
−
* Developers should request changesets get audited and approved before they are pulled into the release branch.  You can tag a changeset for audit by putting '''BP''' on a line by itself in your commit log.  ("BP" stands for "BackPort").  Make sure the '''BP''' is on a line by itself with no surrounding whitespace.  The '''commit-email''' script will make the audit request on your behalf.
 
−
* Any change that gets requested to merge onto a release branch should have an associated entry in [[Bugzilla]].  The bug should contain the changeset number with the bugfix in ''trunk'' (or the name of the bugfix branch).  You should mark this bug as a blocker of [http://bugzilla.gnome.org/show_bug.cgi?id=486922 Bug #486922], which is the catch-all for keeping track of backport requests.  Once the requested and audited change is pulled into the release branch the bug will get closed.
 
−
* If you audit a changeset and believe that it is safe for backporting and does not introduce other bugs (i.e., you've tested it and haven't noticed new issues) then you should add a comment in the bug report and/or respond to the audit-request email.
 
  
 
==Updating Translations==
 
==Updating Translations==
−
 
+
See [[Translation]].
−
Translations are "special".  Translation updates can be made directly into the po directory of the release branch.
+
[[Category:Development]]
 +
[[Category:Translation]]

Latest revision as of 18:49, 24 October 2023

This page describes the development process for developing GnuCash code. It talks about how we use Git and in particular how we use branches to manage the GnuCash source code and release process.

Development Process Goals

The goal of this process is to limit the number of potential bugs that enter the stable release branch(es) due to changes made to the sources. We want to make sure that changes made to releases have been audited and vetted BEFORE they are commited to the release branches. At the same time we want to try to keep the main development branch as stable as practical.

In particular, we want to prevent "bad releases". Those are releases where a bugfix inadvertantly breaks other code.

Documentation

Documentation should ideally start before coding ...

There are 3 target groups for documentation:

Code
For current and future developers add comments in the source files.
API
see Doxygen.
User
has again 2 sources.
In Code
add good tooltips
In the Docs

New or reworked features should also be documented as it turns out, some bugs are a problem of communication between user and developer. If you do not have the resources to update the documentation yourself, you should at least assign a Bugzilla entry to the documentation team with the essential information.

Branches

Branches in Git are cheap, therefore we use branches to our advantage. There are almost always two "interesting" branches in use:

  • stable — the current stable branch leading to then next minor release,
  • future — the development branch leading to then next major release, but only after it differs from stable.

When close to a major release there will be a third "interesting" branch:

  • 'unstable' — the preview branch for the upcoming major release

Older release series are labelled with the release number, e.g. 2.4. Those branches are normally closed to new work shortly after a new stable branch is created from unstable at the beginning of a new release series.

Developers may also have other branches in use for development or bugfixing. This is described later in this document. These will generally not be pushed to the main repository.

Dependencies

Changing dependencies should usually happen on the future branch.

Places to change (minimal) versions
Buildsytem
Code: CMakeLists.txt or
Docs: CMakeLists.txt
README
Code: README.dependencies and
it's wiki derivate Dependencies,
Docs: README
Wiki Templates
Some version numbers are defined in Templates for easier updates of links etc.

Developing New Features

All new development work should target the future branch. This includes new features and enhancements. Small changes can be commited directly to future. Larger features or architectural changes should happen on a feature-branch; once that feature is working it can get merged back into future.

Bug fixes should be built against the stable branch. See Fixing Bugs below for details.

At any particular time, future should be "as stable as possible". We use feature branches to limit the risk that a feature never completes. If development of a feature were done solely in future and that feature never got completed it could leave future in a state where it doesn't compile, or it fails the regression tests.

If you are at all questioning whether to develop directly in future or in a branch, do it in a branch.

As a general rule, development branches should not be pushed back to the main repository. If one needs to publish a development branch, fork the Github repository to your personal account and publish the branch there.

Minor Changes

Developers with the requisite privs may push minor changes directly to stable or to future as appropriate without a feature branch.

Major Changes

Use your judgement about whether your major change should be done directly on future/stable or whether you should use a feature branch. We trust you. Just remember, smaller changesets are easier to audit. When possible try to ensure that each commit can compile; that makes it much easier to bisect when tracking down a regression.

Backwards-Incompatible Schema Changes

To make the policy clearer, backwards-incompatible schema changes in an unstable release, e.g. 5, require a change to the previous stable release, e.g. 4, to provide a read facility for the new schema.

Often it is easier to add a new KVP which will be simply ignored by older versions than to change the schema. But then it would become harder to get the database normalized for the SQL backend.

LibGnuCash API Stability

Historically GnuCash has been developed as a stand-alone library with no concern for stability or backwards compatibility of the core API in spite of simultaneously insisting that the only safe way to operate on GnuCash data is to write code using the somewhat published API in C, Scheme, or Python. Over the first 20 years or so API has mostly been added and seldom pruned, so at as we get into converting more interesting components to C++ it's time to clean up and de-duplicate the API.

As far as we're able to tell no-one has ever written a consumer of the GnuCash C API outside of GnuCash itself, so during the conversion we're not going to worry about it. Everything that isn't Gui or Report will be rewritten in C++ and the C++ API will be new, and once it's internally stable we can begin to treat it like a real library with API guarantees. The C API necessary to interact with the Gtk GUI will remain as long as we retain the latter but there will be no stability guarantee for that part.

The bindings are another matter entirely. Much of the existing C API is also exported to Scheme and Python via SWIG. We know of many cases where users have written custom reports and a few cases of users writing Python extensions. Those users deserve reasonable stability in the API, hence the following policy.

Policy

  • Functions must not change signature (except in C++ which supports overloading). If one needs to do the same thing with different types, make a new function with a different name.
  • Functions may only be deleted in the future branch. Functions to be deleted must first be marked deprecated in the stable branch at least 6 months before the major release.
  • Deprecated functions should be rewritten as wrappers of their replacement functions. If the function is no longer needed in the C library the wrapper should be in the SWIG interface file (e.g. engine.i).

Marking a function as deprecated

  • C: __attribute__((deprecated)) (gcc/clang) or __declspec(deprecated("explanation/redirection string")) (Visual Studio)
  • C++: [[deprecated("explanation/redirection string")]] (Requires C++14, use the C method in stable)
  • Python: warnings.warn("explanation/redirection string", DeprecationWarning, stacklevel=2)
  • Scheme: (issue-deprecation-warning "explanation/redirection string")

Python Note: DeprecationWarnings are disabled by default, so developers of Python programs using GnuCash bindings should test by invoking python with -Wd or setting PYTHONWARNINGS=default in the shell environment.

Fixing Bugs

In general bug fixes should always target the oldest active branch that features the bug. If the bug is present in the current stable version of GnuCash, the bug should be fixed on stable (or a sub branch thereof). If the bug exists only in the development version of GnuCash, the bug should be fixed on future only.

The exception here would be a bug in the stable branch for which the fix itself is a very complex or large body of code. Since this code has a higher risk of introducing new bugs in itself it should not be applied on the stable branch. Instead such a bug can only be fixed on the future branch (unstable version).

Since we frequently merge the stable branch into future, please do a test merge of stable into future in your local repository to ensure that your patch merges cleanly.

If it doesn't and you don't know what to do about it, say so on the bug report when you attach your patch so that a developer can help you asses the situation and decide how to resolve it.

Bugfix Branches

Many times a bugfix is simple. In that case the bug can be fixed directly on the stable branch and the changeset can be merged forward to the future branch so future major releases carry it as well.

If the bugfix is sufficiently complicated then that bugfix warrants a bugfix branch. The developer can create a branch off the release branch (stable) to work on the bugfix, and once that's been tested that branch can be merged back into the release branch.

Release Version Numbering

GnuCash uses two version number schemes:

  • For versions up through 2.7.8, each GnuCash version number consists of 3 numbers separated by a period, in format Fundamental.Major.Bugfix E.g. 2.4.11
  • Beginning with version 3.0, released in April 2018, each version number consists of only 2 integers separated by a period,

meaning Major.Minor, e.g. 3.1.


Fundamental

The only fundamental change, from 1 to 2, involved changing most of the code from Scheme to C and upgrading the GUI from Gtk1 to Gtk2. Since we used this so infrequently we decided to switch to a two-number release.

Major

Major releases are planned every 2 - 4 years. See the Release Schedule for the development teams forecast.

There are 2 types of Major releases, Stable and Unstable.

Stable

Stable releases are intended for use by users with their live data. They have one or two-digit minor versions, e.g. 3.1 or 3.21.

Unstable

Unstable release numbers have three digit minor versions beginning with 900, e.g. 3.902.
Unstable releases are the precursors to a stable release, and are intended only for testing, by either developers or interested community members who don't have the time or skills to build GnuCash from source. They are meant to expose the developers' work to a wide audience, so changes can be tested in a range of environments, before being included in a stable release.
Unstable releases come late in a major development cycle. By the time unstable releases are issued, relevant longer term modifications should be nearly done (except for some bug fixing).
Unstable releases are very short-lived, normally only six months or so, with new releases every month and a final pre-stable release two weeks before the stable release. If a significant problem surfaces in that last release, the developers will focus on that problem and do another pre-stable with the stable to follow two weeks later.
Due to the possibility of data corruption, unstable releases should only be used on a copy of live GnuCash data.

Minor

Minor releases are made typically every 3 months; see the Release Schedule. Minor releases contain fixes to reported problems and small features that the development team consider low risk for incompatibility.

Minor releases should never break file compatibility with other releases in the same major version.

Updating Translations

See Translation.

Retrieved from "https://lists.gnucash.org/wiki/index.php?title=Development_Process&oldid=21926"