Development update (August 11)
And another commit, which only improves the development infrastructure, this time focused on testing.
I removed the
bin/runtests script, which collects and runs all tests, and moved that logic into
waf, so tests are now run with the command
./waf test. The main benefit is that I'm getting
parallel execution almost, but not quite, for free.
The drawbacks are:
- Each test module is executed in a subprocess, which causes some overhead. That's specifically noticeable for the unittests, where the tests themselves are fast and thus the overhead matters more.
pylintis now also run as a subprocess.
runtests.pyimported it as a
pythonmodule and subsequent
pylintruns could use previously cached data.
- I needed some additional code to store the results of each tests and merge them at the end. For
the unittests I'm using
unittest-xml-reportingto write them out as XML and
xunitparserto read them back. For
pylintI'm just writing/reading plain text files.
mypyneeded some extra care, because running multiple
mypyprocesses using the same cache directory causes some race conditions. I have to use a pool of caches, so each cache is only used by one
mypyprocess at a time.
But overall the test suite now runs about twice as fast. Some unscientific benchmarks:
I just used a single run of each command and
/usr/bin/time for the measurement.
The overhead is about 2x, so you need two cores just to make of for it. Which means that you're penalized, if you are attempting to do noisicaä development on a single core machine. But I guess using a single core machine (which must be pretty old) isn't much fun anyway these days.
What is interesting is that there is little gained when going from four to eight cores. That's probably because my CPU is a quad-core with 2x hyper-threading, but I haven't looked into that issue any deeper.
Another advantage of running tests in subprocesses is that I can now put a timeout on tests, so if some test hangs, it will eventually fail and I don't have to kill the main process, thus losing the actual test report.
I reanimated the VM tests, which suffered from some bitrot and didn't work anymore. While at it, I
qemu for the VM, because
qemu is a bit easier to automate.
These tests launch a VM with a minimal installation of the OS (current only Ubuntu 16.04 and 18.04) and then build noisicaä from the sources. This is mostly to verify that all dependencies are correctly declared and the build instructions work as advertised on a system that isn't my development system.
Once I got the tests working again, they uncovered some bitrot, which caused noisicaä to not work anymore on Ubuntu 16.04 or Python 3.5.
The test suite now runs
C++ source files. I previously cranked up the pickiness
-Werror -pedantic), but that meant that compilation would fail for every minor
issue. Now building became more "pythonic":
C++ source get built, as long as there are no major
issues (with all warnings disabled, i.e.
-w), and once I want to also have the code "clean" I run
clang-tidy over it.
I haven't verified, if the issues that
gcc would have detected are all covered by
clang-tidy. I'm just assuming that
clang-tidy is "good enough". I also have not yet attempted to
clang-tidy and run it with default settings for now.
mypy to 0.720
That version specifically has a new option
warn_unused_ignores, so I could find and remove
overrides, which used to fix false-positives in some previous version of
mypy, but were now
obsolete and potentially masked some real issues.
Removed all build noise
At least with my setup, all random noise generated during the build steps has been removed.<<< waf migration