Nix Builder design

Introduction

kernel-builder uses a Nix-based builder that orchestrates the build. The Nix builder provides:

• Reproducible evaluation. The same Nix builder version will always produce the same derivations (build recipes).
• Largely reproducible builds by using a build sandbox that only has the dependencies specified in a derivation.
• Seamless creation of different build environments (e.g. different Torch and CUDA combinations).

Kernel build steps

A kernel derivation builds a kernel in the following steps:

1. Generate CMake files for the kernel using kernel-builder create-pyproject.
2. Generate Ninja build files using CMake.
3. Build the kernel using Ninja.
4. Perform various checks on the compiled kernel, such as:
   • Verify that the kernel only uses ABI3/manylinux_2_28 symbols.
   • Verify that the kernel can be loaded by the kernels Python package.
5. Strip runpaths (ELF-embedded library directories) from kernel binaries to make the kernel distribution-independent.

manylinux_2_28 compatibility

To achieve manylinux_2_28 compatibility, kernels are built using a toolchain similar to the manylinux_2_28 Docker images. This toolchain is based on the gcc toolsets from AlmaLinux 8. manylinux_2_28 uses AlmaLinux 8 as its base, so we have to compile against the same glibc/libstdc++ versions to ensure compatibility.

We repackage the AlmaLinux 8 toolsets and libstdc++ as Nix derivations (see the nix-builder/packages/manylinux_2_28 source directory). Then we merge various toolset packages to an unwrapped gcc that resembles unwrapped gcc in nixpkgs. Finally, we wrap binutils and gcc to combine them into a stdenv.

The stdenv does not reuse glibc from AlmaLinux, since its dynamic loader has hardcoded FHS paths (/lib64 etc.) that are not valid in Nix. Using this dynamic loader results in linking errors, since the paths in the dynamic loader are used as a last resort (to link glibc libraries). So, instead we build our own glibc 2.28 package (see nix-builder/pkgs/manylinux_2_28/stdenv.nix) and use that.

The package set pattern

We repackage various existing package sets as Nix derivations. For instance, this is done for ROCm, XPU, and manylinux_2_28 packages. We do this because we want these libraries to be as close as what the user would install. This avoids compatibility issues between the kernels and the official vendor packages. For instance, suppose that we built a ROCm library as a shared library and ROCm provides the same library as a static library, then compiled kernels could use symbols that cannot be resolved when installing the official ROCm packages. Similarly, using the official packages allows us to test against the official upstram packages.

These package sets all follow the same pattern:

{
  lib,
  callPackage,
  newScope,
  pkgs,
}:

{
  packageMetadata,
}:

let
  inherit (lib.fixedPoints) extends composeManyExtensions;

  fixedPoint = final: {
    inherit lib;
  };
  composed = lib.composeManyExtensions [
    # Base package set.
    (import ./components.nix { inherit packageMetadata; })

    # Package-specific overrides.
    (import ./overrides.nix)

    # Additional overlays that extend the package set.
    (import ./some-overlay.nix)
  ];
in
lib.makeScope newScope (lib.extends composed fixedPoint)

We use a fixed point to build up the package set as a list of overlays. This has various benefits. For instance, it allows us to refine the package set incrementally and we can refer to the final versions of packages in intermediate overlays.

The package sets all use a similar list of overlays:

• An initial overlay (components.nix) that applies a generic builder to the package set metadata. The metadata typically comes from a Yum/DNF repository that contains RPM packages.The generic builder will extract the RPMs and move binaries, libraries, and headers to the right location. This results in a set of Nix derivations that may or may not build.
• The next overlay (overrides.nix) fixes up derivations generated by the generic builder in the previous overlay that do not build. Fixing the derivations typically consists of adding missing dependencies and changing embedded FHS paths to Nix store paths.
• Additional overlays with derivations that combine outputs from previous overlays. One typical example are derivations that construct a full compiler toolchain (e.g. nix-builder/pkgs/manylinux_2_28/gcc-unwrapped.nix).