waf migration

I could not resist and spent the last few days converting the build system from cmake to waf.

Surprisingly the resulting commit has a negative line count, even though the new system has some new features.

I was never really happy with cmake. I needed a proper build system, because noisicaä isn't a pure python application (which most of the other things are, that I do in my spare time). For the initial prototype I got away with using setup.py to build the few cython extensions, but I quickly grew out of that. My usual choice would have been good, old make, but one of the common patterns of noisicaä development is to upgrade my toolbox and try something more modern. cmake advertises itself as modern, fast and flexible, and it's used by a lot of large open source projects. So I decided to give it a try.

And it's certainly easy to use for c++ projects - everything needed for that is built-in. But my case involved a mix of python, cython as well as a bunch of custom build rules to auto generate various files, e.g. using csound, faust, etc. To define such custom build rules, you have to dive deep into cmake's language and that's the part that turned me off.

cmake's language feels like a reanimated corpse that dies a few decades ago. It's a macro language, there are no proper functions, so "return values" are written to some kinda global variables. While you can get everything done somehow, it feels really quirky.

Also cmake's distinction between files and targets was very confusing to me. Only target can have dependencies on other targets - at least that's how I understood it. I could never really wrap my head around it, so I have probably created a lot more targets than necessary.

waf is not without it's own quirks. Part of that might be due to compatibility requirements. On the one hand compatibility with older versions of waf, i.e. the usual accumulation of cruft that you see in any non-trivial application. On the other hand compatibility with old python version. E.g. waf's Node class looks a lot like pathlib, but it cannot use it as long as it wants to stay compatible with python versions before 3.4. And because it wants to be both self contained and small, it can't just use packages outside of the stdlib.

But the biggest selling point of waf is that it's python. Instead of some half baked custom language, you have the full power of a proper general purpose language at your fingertips.

Converting the existing build logic over to waf was pretty painless. Basically just rename all CMakeLists.txt files to wscript and make some fairly mechanical syntax changes. Most of the work was reimplementing the various custom rules to waf. That turned out to be a bit more verbose in terms of line count, but that's more than compensated just by expressing the logic in a proper programming language.

The only significant difference should be that I never got dependencies of cython modules right in cmake. Every once in a while I made a change that should have triggered a rebuild of some cython modules, but that never happened - resulting in runtime errors. Getting that right in waf was very straight forward.

Once I did the basic build logic carried over to waf, I started looking into things I could do better now.

Handling of 3rd party dependencies

One of the things that neither cmake nor waf (nor any other build system that I know of) directly support, is the handling of dependencies on 3rd party packages. That's actually something that could be done with setup.py, which is more of a package management system than a build system - at least for other python packages (and with a terrible hack, also for non-python libraries).

Before you can start to build any software from source, you probably need a bunch of libraries, packages, tools, etc. installed, which are used by those sources. Usually the configure step of the build system checks for these dependencies and fails if anything is missing. It is then left to the user, possibly directed by some INSTALL documentation, to install the missing pieces and try again. That usually takes a few iterations until everything is in place. That's pretty tedious for the user, and it's hard to keep the documentation in sync with the actual requirements of the software.

I used to have a listdeps script, which had all the knowledge about required packages, both python packages that are to be installed with pip, as well as system packages to be installed with apt. This way the list of required packages was encoded in a way that did not need human intelligence to decipher. I have fully automated tests, which perform the installation on a minimal system (running in a VM), thus ensuring that the list of requirements is complete.

I now took this approach a step further and integrated it directly into waf configure. So waf checks if all required packages are present and if not, simply installs them. python packages are downloaded and installed in a virtual environment. System packages are installed with apt, which might trigger a password prompt. Usually the user does not expect configure to actually modify the system, so that behavior must be enabled by some flags.

There were a few dependencies (e.g. csound, protoc, ...), which are not available as easily installable packages - usually because I want a newer version that what is available in Ubuntu. For those packages I used that terrible hack I mentioned above. For each such dependency I had a setup.py file, which I could install with pip. But instead of having any real sources of its own, those setup.py scripts just had the logic to download, build and install the sources of those packages into the virtual environment. I have now moved that logic directly into waf, but apart from that I'm following the same approach: instead of documenting that you should download and install library X at version Y, the build system just does it itself.

Transparent virtual environment handling

As usual for python projects, I'm using a virtual environment to locally install 3rd party python packages. And I go one step further and also install locally built versions of csound, protoc, etc. there as mentioned above.

The normal process is to first "activate" the virtual environment, so these packages become visible to python.

I now moved the management of the virtual environment into waf, so it becomes completely transparent to the user. It is created and populated by waf configure and automatically activated by subsequent uses of waf or when running noisicaä from the build directory.

waf install

Just out of curiosity I tried out what happens when I run waf install. And it turned out that the changes to get that work correctly weren't that difficult. The trickiest part was again dealing with those 3rd party packages. For noisicaä to work, it needs those packages and libraries installed with its own files. I ended up just copying those files from the virtual environment into the target lib directory.

<<< Development update (July 14)