Using Build-Actions to Simplify CI Workflows

Continuous integration (CI) is one of the most important tools to ensure that your project compiles, passes tests, and is usable in a variety of operating systems and runtime environments. Of course the matrix of environments used to test a particular project depends on its nature, however, it's very simple to write code that only compiles on Linux, MacOS, and Windows, but not on Android or variety of BSDs, for example.

In this post, I would like to explain why I have created build-actions and how it helps me to maintain CI pipelines (GitHub actions at the moment) that build and test projects that I have started and continue to maintain. I'm actually not suggesting to use build-actions in your own project, just that having code shared between multiple projects could decrease the time required to manage their CI pipelines. The environments that you target change, and with these changes CI pipelines have to be changed as well, so the goal is to minimize the time required to change the workflows.

Build Matrix

The first thing that I would like to talk about is a build matrix. This is essential for every CI pipeline as usually testing in a single environment doesn't ensure that the project works in others. The number of properties of a build matrix depends on a project, but in the C/C++ world it basically means the following:

  • Environment (OS / Runtime):
    • Host - host operating system or runtime where the CI runs (only useful for cross-compilation).
    • Target - target operating system or runtime where the compiled product would run, in many cases Host OS == Target OS (note that Emscripten would be considered a runtime and not OS, but it's still a valid Target).
  • Target architecture - X86, ARM, RISC-V, JS/WASM, ...
  • C/C++ Compiler - gcc, clang, msvc, ...
  • Build Options - build mode (debug/release), build options, sanitizer options, ...
  • Project Dependencies:
    • Version - each dependency has a version (multiple versions of the same dependency explode the build matrix).
    • Build Options - non-header only dependencies must be built, so they include additional build options.
  • (plus extra stuff depending on a project)

Problems with build matrix that usually lead to duplicated steps in workflows:

  • Every OS offered by GitHub will once be deprecated. There are os-latest versions, but these will have different tools when the OS is upgraded, which means that a specific compiler version that a matrix was using before could no longer exist. This means that it's usually better to depend on a specific OS version and do the upgrade manually when a new version is provided, which implies a maintenance cost.
  • Not every OS and architecture can be part of a default build matrix. If you need to test a different OS or architecture than offered by GitHub actions you have to use virtualization, and virtualized workflows have to be much simpler than native ones.
  • Installation of build dependencies may vary depending on OS and its version. For example on Linux apt-get can be used on a variety of Linux distributions (but not on all of them), but for example every flavor of BSD has a bit different tooling.

This is not a complete list, just the most painful issues that every maintainer of a CI pipeline building a C/C++ project has most likely faced.

Pipeline Jobs

Let's use the term job to describe the steps that every combination of a build matrix has to run. Each job will most likely look like this:

  • Checkout the source code that the job will build and test.
  • Prepare the environment, which means installing all tools needed to build the project (compiler, cmake, scripting languages, ...).
  • Fetch the necessary dependencies that the project needs, which is typically done after you have checked out the project and all the tooling as fetching the dependencies may require the tools (be it python, conan, cmake, ...).
  • Prepare building the project (configure), which could basically mean using cmake to create makefiles or ninja files, ...
  • Build the project (make, ninja)
  • Test the project
  • Post-build steps (upload artifacts, deploy, ...)

In general, many of these steps will be very similar between multiple projects. Setting up the environment and preparing the C/C++ compiler that we want to test could mean using apt-get to install the compiler, etc... But that's just not all of it. Everyone who ever used GitHub actions knows that if you run apt-get 1000 times on a GitHub runner it will fail few times because of a connection timeout. When a build matrix grows and you have for example 50 jobs having various configurations to run on a Linux runner, the chance of a failed build because of a connection timeout grows with the nunber of jobs.

In a smaller project or a project that uses a very simple CI pipeline, you can just re-run the failed job and be happy if it finishes. However, if you have to re-run some job every day or two because of a connection timeout then it becomes annoying. The problem is, where would you want to put the logic to handle this case and to repeat the apt-get call? I have decided that I don't want to implement this per project as I have multiple projects and syncing this would cost me a lot of time.

Build Actions

build-actions is a project that I have created to maintain parts of CI pipelines that I don't want to duplicate. In other words, it helps me to move all the annoying parts of workflow away from projects to a common place, where I can implement all the ugly parts without having to maintain them separately per project.

I have decided to use Python 3 to implement build-actions, because I can debug the code locally before even using it in a workflow. In addition, if I need to implement some workaround (like the apt-get timeout problem) then it's possible to implement it in build-actions instead of infecting workflows with that. This means that historically I have been able to fix the same CI problems that happened in multiple projects by fixing build-actions instead of touching workflows of these projects.

The following problems can be solved by build-actions:

  • Installing python - this is only necessary when running inside a VM where you cannot use setup-python action.
  • Installing build tools - this includes tools like cmake, C/C++ compiler, and extra packages that may be required by sanitizers or other tools. In addition testing 32-bit X86 binaries on 64-bit Linux requires additional (multilib) packages.
  • Using problem matchers - both VS and gcc/clang matchers are provided, so you don't have to pull these into your projects.
  • Configuring the project - calling cmake with options taken from a build matrix (compiler, build options, diagnostics, sanitizers, ...).
  • Building the project - also uses cmake under the hood.
  • Testing the project - it doesn't use cmake test for this, instead it reads a JSON file that provides some workflow parameters including how to test the final product. The reason for this is that by using this the tool can run the same test with various parameters, for example.

Since I don't want to copy-paste sample workflows into this post, I offer links to the most important projects that use build-actions:

As you can see from the workflow files, a single build matrix is used to test each library on Windows, Linux, and MacOS. Each OS is additionally tested with multiple compilers and build options, which are also part of the build matrix. Also, there are workflows that use sanitizers and other tools such as valgrind. This all in a single build matrix using the same steps that just invoke build-actions/action.py command.

But this is not all - check out build-vm job that uses cross-platform-actions to run builds in a VM. When a build is running in a VM it means that you get a minimal VM with some OS and you have to perform all the setup by yourself (there is no way to use existing actions in such VM). This means no checkout or setup-pythonaction is available. That's the reason why build-actions implements a preparation step to even setup python on supported environments as it greatly simplifies such workflows. And the build steps in such VM are just the same, but instead of running four steps after each other --step=all is used to make the workflows simpler.

Conclusion

build-actions project simplified all workflows in projects that I have started and maintain. This little tool has allowed me to test these projects on a variety of operating systems with a variety of C/C++ compilers without forcing me to duplicate all that effort in every project. In addition, all the projects properly use ASAN, UBSAN, scan-build, and valgrind in every build, which I would recommend to do if you take your project seriously. Each of these tools is integrated by having one extra line in a build matrix.

If you think that "my code compiles on Linux and MacOS just fine, so it would compile on other BSDs too"; - well, maybe... Or maybe somebody will open Issue #399 regarding a broken NetBSD support. The reason is that

"what isn't tested is broken, but you don't know it yet"

at least that's my own experience when it comes to software development.

This post is not a promotion of build-actions tool at all. If there was anything better I would gladly use it so I don't have to maintain my own tool. What I'm trying to say is that all available tools should be used in CI workflows, because they catch problems that could otherwise get unnoticed.