Continuous integration (CI) is one of the most important tools to ensure that your project compiles, passes tests, and is usable in a variety of operating systems and runtime environments. Of course the matrix of environments used to test a particular project depends on its nature, however, it's very simple to write code that only compiles on Linux, MacOS, and Windows, but not on Android or variety of BSDs, for example.
In this post, I would like to explain why I have created build-actions and how it helps me to maintain CI pipelines (GitHub actions at the moment) that build and test projects that I have started and continue to maintain. I'm actually not suggesting to use build-actions in your own project, just that having code shared between multiple projects could decrease the time required to manage their CI pipelines. The environments that you target change, and with these changes CI pipelines have to be changed as well, so the goal is to minimize the time required to change the workflows.
The first thing that I would like to talk about is a build matrix. This is essential for every CI pipeline as usually testing in a single environment doesn't ensure that the project works in others. The number of properties of a build matrix depends on a project, but in the C/C++ world it basically means the following:
Problems with build matrix that usually lead to duplicated steps in workflows:
os-latest versions, but these will have different tools when the OS is upgraded, which means that a specific compiler version that a matrix was using before could no longer exist. This means that it's usually better to depend on a specific OS version and do the upgrade manually when a new version is provided, which implies a maintenance cost.apt-get can be used on a variety of Linux distributions (but not on all of them), but for example every flavor of BSD has a bit different tooling.This is not a complete list, just the most painful issues that every maintainer of a CI pipeline building a C/C++ project has most likely faced.
Let's use the term job to describe the steps that every combination of a build matrix has to run. Each job will most likely look like this:
In general, many of these steps will be very similar between multiple projects. Setting up the environment and preparing the C/C++ compiler that we want to test could mean using apt-get to install the compiler, etc... But that's just not all of it. Everyone who ever used GitHub actions knows that if you run apt-get 1000 times on a GitHub runner it will fail few times because of a connection timeout. When a build matrix grows and you have for example 50 jobs having various configurations to run on a Linux runner, the chance of a failed build because of a connection timeout grows with the nunber of jobs.
In a smaller project or a project that uses a very simple CI pipeline, you can just re-run the failed job and be happy if it finishes. However, if you have to re-run some job every day or two because of a connection timeout then it becomes annoying. The problem is, where would you want to put the logic to handle this case and to repeat the apt-get call? I have decided that I don't want to implement this per project as I have multiple projects and syncing this would cost me a lot of time.
build-actions is a project that I have created to maintain parts of CI pipelines that I don't want to duplicate. In other words, it helps me to move all the annoying parts of workflow away from projects to a common place, where I can implement all the ugly parts without having to maintain them separately per project.
I have decided to use Python 3 to implement build-actions, because I can debug the code locally before even using it in a workflow. In addition, if I need to implement some workaround (like the apt-get timeout problem) then it's possible to implement it in build-actions instead of infecting workflows with that. This means that historically I have been able to fix the same CI problems that happened in multiple projects by fixing build-actions instead of touching workflows of these projects.
The following problems can be solved by build-actions:
cmake test for this, instead it reads a JSON file that provides some workflow parameters including how to test the final product. The reason for this is that by using this the tool can run the same test with various parameters, for example.Since I don't want to copy-paste sample workflows into this post, I offer links to the most important projects that use build-actions:
As you can see from the workflow files, a single build matrix is used to test each library on Windows, Linux, and MacOS. Each OS is additionally tested with multiple compilers and build options, which are also part of the build matrix. Also, there are workflows that use sanitizers and other tools such as valgrind. This all in a single build matrix using the same steps that just invoke build-actions/action.py command.
But this is not all - check out build-vm job that uses cross-platform-actions to run builds in a VM. When a build is running in a VM it means that you get a minimal VM with some OS and you have to perform all the setup by yourself (there is no way to use existing actions in such VM). This means no checkout or setup-pythonaction is available. That's the reason why build-actions implements a preparation step to even setup python on supported environments as it greatly simplifies such workflows. And the build steps in such VM are just the same, but instead of running four steps after each other --step=all is used to make the workflows simpler.
build-actions project simplified all workflows in projects that I have started and maintain. This little tool has allowed me to test these projects on a variety of operating systems with a variety of C/C++ compilers without forcing me to duplicate all that effort in every project. In addition, all the projects properly use ASAN, UBSAN, scan-build, and valgrind in every build, which I would recommend to do if you take your project seriously. Each of these tools is integrated by having one extra line in a build matrix.
If you think that "my code compiles on Linux and MacOS just fine, so it would compile on other BSDs too"; - well, maybe... Or maybe somebody will open Issue #399 regarding a broken NetBSD support. The reason is that
"what isn't tested is broken, but you don't know it yet"
at least that's my own experience when it comes to software development.
This post is not a promotion of build-actions tool at all. If there was anything better I would gladly use it so I don't have to maintain my own tool. What I'm trying to say is that all available tools should be used in CI workflows, because they catch problems that could otherwise get unnoticed.