# ADR 7 — ZFS on data disks Status: accepted ## Context Some nodes keep local state on their data disks, for example a persistent-volume provisioner, an image or build cache, or scratch space. These nodes want what ZFS gives and `raid[]` does not: end-to-end checksums, transparent compression, and snapshots. [Issue](https://github.com/Corium-OS/Corium/issues) asked whether Corium could put ZFS on those data disks. The request splits into two questions, as software RAID did (see [ADR-0003](/docs/decisions/adr-0003-software-raid-scope/)). **Data disks.** These are extra disks that you pool and mount somewhere. This can happen at first boot, and this ADR adds it. **The root filesystem.** When something reads the `corium:` block, root is already deployed and mounted, and the process that reads the block runs from it. At that point, nothing can move root onto ZFS. Root on ZFS also needs the module in the initramfs and a bootloader integration that has no declarative equivalent in bootc. This is out of scope, for the same reasons as a RAID root. But ZFS has a cost that RAID does not have, and the image pays it, not the schema. - **Fedora ships no ZFS.** ZFS is under the CDDL, and Fedora treats the CDDL as incompatible with the GPL kernel. No package exists to install. You must compile the module from source. - **Fedora has no stable kABI.** A module built against one kernel will not load on another. Unlike RHEL, Fedora has no kmod that tracks the kABI. You must build the module against the *exact* kernel that it runs on, and build it again for each new kernel. ## Decision `zfs[]` covers data disks. `corium-agent` creates the pools at first boot, before it installs k0s, with the same rules as `raid[]`: - It refuses devices that already hold data. - It adopts an existing pool and does not rebuild it. - A `RequiresMountsFor` drop-in stops k0s from starting before the datasets are mounted. The ZFS kernel module and userspace are **not in the base image**. They ship in a derived variant, `deploy/zfs/`, which you build when you need it. The appliance image has the same shape. The agent code (`internal/bootstrap/zfs.go`) is in the base binary. Only the module and the tools that the code drives are in the variant only. ## Why the module is baked, not built at runtime On an immutable OS, the kABI problem has exactly one clean answer, and bootc already implies it. Build the module against the kernel of the image, at image build time, and ship the two together. An image has one kernel. The module in the image is the module for that kernel. An upgrade replaces both at the same time, by digest. The running kernel and its ZFS module never disagree, because they never update separately. DKMS rebuilds the module on the running machine after each kernel update. It is the answer for a mutable host, and it is wrong here, for two reasons. First, root is read-only, so no place exists to build. Second, DKMS keeps its state under `/var`. On a bootc node, `/var` is machine state that the install seeds once, not a build tree. OpenZFS itself recommends precompiled kmod packages over DKMS for fleets. So the variant compiles OpenZFS into a `kmod` RPM in a temporary build stage and installs it. The module goes into `/usr/lib/modules//extra`, which is part of the read-only image. ## Why a variant image, not the base image The base image is minimal on purpose (see the package list in the `Containerfile`). ZFS would add three things that a node which never uses ZFS must not carry: - **Weight.** The module and userspace add size to an image that exists to be small and replaceable. - **Build fragility.** Each new kernel becomes a compile, and the compile can fail when OpenZFS does not yet support the new kernel. If that compile gated *every* Corium build, the release cadence of the whole project would depend on the kernel support of OpenZFS. As a variant, it gates only the nodes that opted in. - **A licensing choice.** To compile and ship a CDDL module in a GPL kernel image is a decision that an operator must make on purpose. It must not be baked into the artifact that everyone downloads. The variant derives `FROM` the base image, and this makes the kABI match free. `rpm -q kernel` in the build names the exact kernel that the module must target. Thus the compile uses the kernel that the image already ships. ## Why data-disk ZFS earns its place over cloud-init The reasons are the same three as for `raid[]` (see ADR-0003): - **Ordering.** A `runcmd` that calls `zpool` races the kubelet. If it loses the race, containerd writes into the directory on the root disk. Then the dataset mounts over that directory. After that, the data is invisible but still uses space on root. Corium creates pools before k0s and writes `RequiresMountsFor`. Thus Kubernetes refuses to start instead of running without its storage. - **Refusing to destroy data.** If a device already has a filesystem, a partition table, or the label of another pool or array, the bootstrap stops. `zpool create` by hand takes `-f` and does not ask. Corium makes `wipe: true` the explicit consent, and by default it stops. A device can belong to a pool or to an array, never both. Validation makes sure of that. - **Idempotency.** A second bootstrap imports and adopts an existing pool and does not create it again. It reconciles dataset properties and does not duplicate anything. ## Consequences The data pool of a node survives a reboot. The node imports it by a scan of the attached disks, with `cachefile=none`, so the pool follows its disks onto replacement hardware. The pool does not survive the loss of the node. The rest of Corium uses the same model. If you declare `zfs[]` on a plain base image, first boot fails with an error (`loading the zfs kernel module: ...`). It does not silently do nothing. A request for a pool that the image cannot build must fail loudly. The build of the variant follows the kernel support of OpenZFS. If a new Fedora kernel is newer than what the pinned OpenZFS release supports, the fix is a new `ZFS_VERSION`, not a change to the base image. If Fedora ships ZFS, or bootc adds a declarative root-on-ZFS path, revisit this decision. The schema and the variant both have room. ## Alternatives considered **ZFS in the base image, behind a build ARG.** A single `Containerfile` with a conditional kmod stage. Rejected, because it makes the one file that every build reads more complex, for a feature that most nodes do not use. Also, a stage-selection trick that skips the compile when the ARG is off is exactly the kind of cleverness that the base image avoids. A separate `deploy/zfs/` follows the example of the appliance image and keeps the base `Containerfile` easy to read from top to bottom. **DKMS on the node.** Rejected above: no writable root exists to build on, and `/var` is seeded state, not a build tree. **Document a `zpool` recipe, add no field.** Rejected for the reason that ADR-0003 rejected the same option for RAID: the ordering problem stays unsolved. That is the problem that corrupts a node without any warning. **Put the data-disk provisioner on the RAID path instead.** `raid[]` already exists and covers striping and mirroring. Rejected as a *replacement*, but kept as an alternative. mdadm gives a block device, and you put a filesystem on it. It has no checksums and no snapshots, and the workloads that want ZFS want exactly those. The two coexist. A node chooses one for each set of disks.