ADR 3 — Software RAID scope

Status: accepted

Context#

Issue #5 asked for a way to declare a software RAID array from the corium: block. It added an explicit condition: add the feature only if it is better than a plain cloud-init disk setup.

The one request contains two separate problems.

Spare disks. These are extra disks on a node, which you mirror or stripe and mount somewhere. This can happen at first boot.

The root filesystem. When something reads the corium: block, the root filesystem is already deployed and mounted. The process that reads the block runs from it. At that point, nothing can move the root filesystem onto an array. The tool that wrote the disk must set up a redundant root, and that tool is the installer.

Decision#

raid[] covers spare disks. corium-agent creates the arrays at first boot, before it installs k0s.

The documentation describes a RAID root filesystem as a Kickstart procedure at install time. We verified this procedure on hardware. The schema does not model it.

Why spare-disk RAID earns its place#

cloud-init has no RAID support. disk_setup makes partitions and filesystems. For years, the documentation said that mdadm support was planned. Thus the real alternative is a runcmd block that calls mdadm. Three things make that worse than a field:

• Ordering. runcmd races the kubelet. If runcmd loses that race, containerd writes its state into the directory on the root disk. Then the array mounts over that directory. After that, the data is invisible but it still uses space on the root disk. The node reports a full disk with no visible cause. Corium builds arrays before it installs k0s. It also writes RequiresMountsFor into the k0s unit. Thus Kubernetes refuses to start instead of running without its storage.
• Refusing to destroy data. If a device already has a filesystem, a partition table, or the metadata of another array, the bootstrap stops. mdadm --create by hand does not ask.
• Idempotency. A second bootstrap adopts an existing array and does not rebuild it. It rewrites its own fstab line and does not append another one.

Why a RAID root is not a schema field#

The reason is not that it fails. It works. We verified a bootc install onto RAID1 / and RAID1 /boot on a machine with two disks. We detached the first disk and booted from the second. The arrays were degraded and still served data.

It is not a field because a field would give a false idea of when the decision happens. corium-agent reads the corium: block at first boot. At that point, the root filesystem is already deployed and mounted, and the process that reads the block runs from it. A field can do nothing at that moment.

You must also do three things by hand at install time. Anaconda does none of them for you. docs/raid.md documents all of them:

1. Build a second ESP on the mirror disk and register it with the firmware. Anaconda creates one ESP and warns that a drive failure will make the system unbootable. The warning is correct.
2. Do that in %post with disk-level tools only. Files that %post writes into the /etc and /var of the deployment do not survive a bootc install.
3. Add nofail to the /boot/efi line in /etc/fstab. Without it, the mount is RequiredBy=local-fs.target. If the first disk fails, the machine goes into emergency mode, although the mirrored filesystems are fully healthy.

Fedora CoreOS does all three with boot_device.mirror in Ignition. It builds the array in the initramfs and copies /boot to each disk. bootc has no equivalent. Until bootc has one, this belongs in a procedure, not a field.

Consequences#

If the root disk of a node fails, you reprovision the node. You do not repair it. The rest of Corium uses the same model: the OS is an image, the node is replaceable, and a replacement boots the same digest.

Spare-disk arrays cover the case that actually costs much: local state that is slow to rebuild. The cluster covers the loss of a whole node.

If bootc adds a declarative equivalent of boot_device.mirror, revisit this decision. The schema has room for it.

Alternatives considered#

Documentation only, no field. Ship the mdadm recipe and add no permanent key. Rejected, because the ordering problem stays unsolved. That is the problem that corrupts a node without any warning.

Support a RAID root via Kickstart as a product feature. Rejected for now, but for a narrower reason than “it does not work”. It works, and the procedure is documented. But it is not declarative. It needs a hand-written kickstart, a %post block, and an edit to fstab after the install. It also applies only to the ISO path. A Corium field for it would promise a uniformity that the install paths do not have.

bootc-generic-growpart.service also always fails on an md root. It reads /sys/class/block/<name>/partition, and an md device does not have that file. The failure causes no damage. But every such node reports degraded, and that is a bad result to ship as a supported configuration.

If bootc adds a declarative equivalent of boot_device.mirror, revisit both this alternative and the decision above.