Automated win32 build scripts and UPDATE_SOURCES, packaging dir, ...

[John Ralls]

On Aug 11, 2012, at 8:29 AM, Geert Janssens <janssens-geert at telenet.be> wrote:

> I have just added some scripts to run an automated build from git instead of svn.
> 
> I copied the svn behaviour as closely as possible. While doing so I got to question parts of the general setup we promote [1]. The guidelines state that the contents of packaging/win32 should be copied locally, and it shouldn't be in the repo checkout that will later be used for building.
> 
> I see several issues with this, some of which may be easier to fix than others:
> - I first wondered why the packaging directory should be separate from the repo used to build. After some pondering I came up with this: when UPDATE_SOURCES=yes, and we're using the packaging dir from the repo, the install scripts may update from under us while building. Obviously not good.
> - So I started thinking about UPDATE_SOURCES. Why is that option in there ? I suppose for the "automated" mantra. Having it in there, allows you to always build the most recent state of a branch. I wonder though if this benefit outweighs the restrictions and limitations it brings with it.
> - Copying the packaging directory is a bootstrap thing and has other issues: the guidelines don't explain how to copy this directory. On a clean Windows system your option is to manually download each file or use some kind of download manager. At this stage the user isn't supposed to have svn or git available to checkout the directory - that means the guidelines don't ask him to do so, because it will be installed later on "automatically". This looks like a hurdle to me for new potential contributors on the Windows platform (there are many more, I know...)
> - Next, by simply copying the files to a local directory, the user looses the main advantage a checked out repo has: easy update to newer versions of the build scripts. The user has to guess if/when the scripts were updated and manually copy the over again from somewhere. So if the user isn't careful, later builds may be using outdated build scripts.
> - The fix for the last problem would be to use some form of checkout of the packaging directory. But at bootstrap time, there's no subversion or git available. That can be solved by asking the user to install either tool manually.
> - For svn, the story ends here. The user uses svn to checkout the packaging directory and continues the setup. To get the most recent build scripts, it suffices to run svn update.
> - But for git users, there's another catch: git doesn't allow to checkout one directory only from a git repo. So to satisfy the original requirement (separate packaging directory) he needs to clone the complete git directory. The build itself will clone it once more. That's twice the disk space for the complete gnucash repository used for only ten or something scripts.
> 
> And all that because UPDATE_SOURCES risks of invalidating the build scripts mid-build.
> 
> I see two possible fixes:
> - drop UPDATE_SOURCES. It's not that hard to update your build repo for either svn or git. For git which is much more branch focussed than svn, I think an active developer would probably disable UPDATE_SOURCES anyway, because it would rarely do what you actually wanted. With UPDATE_SOURCES out of the way, the bootstrap procedure would be something like: install your favourite revision control tool, check out the proper repository, (the other bootstrappers, such as mingw-dtk,...), cd into packaging/win32 and off you go. This could still work with building from tarball as well.
> - alternatively if UPDATE_SOURCES is to stay, we could also move the packaging/win32 directory to a separate repository (similar to what the OS X build does). That would allow UPDATE_SOURCES to remain and have a local packaging directory that is revision controlled from the start. Doesn't make much difference for svn users, but for git users the separate packaging repository would be a small clone instead of having to checkout the whole gnucash repository. So for this solution the steps would be: install your favourite revision control too, checkout the packaging repository. Do the other bootstrapping stuff and off you go. The packaging repo can be updated whenever needed.
> 
> Personally I prefer the first option. From a new user's point of view it would look so much more intuitive to be able to either install a tarball or the preferred rcs and checkout the proper repo and from there on start building (I deliberately ingore some other bootstrap steps here to create the picture). But the second option would work as well for me.
> 
> What do others think ? Is my reasoning correct ? What do you see as solutions ?

There are two other options that come to mind:

1. A bootstrapping script can use wget to retrieve the files it needs from https://raw.github.com/Gnucash/gnucash/<branch>/packaging/win32/foo, no need for downloads by hand.

2. Switch to jhbuild if it can be made to work. Sam Thursfield did some work on this which I think Fréderic Peters has committed into the main repo. See https://live.gnome.org/Windows/MingwNativeBuild.

But more generally, yes, UPDATE_SOURCES gets in the way of an active developer, so it should probably be off by default anyway.

Regards,
John Ralls