Software RAID

Corium can build software RAID arrays from the spare disks of a node. You declare them in the same corium: block as all other configuration. To put the root filesystem on RAID is a different problem with a different answer. This page covers both.

Two problems, not one#

This split is not a design preference. It is a consequence of the boot order. When the node reads the corium: block, the root filesystem is already deployed and mounted. The process that reads the block runs from it. At that point, nothing can move the root filesystem onto an array. Only the tool that first wrote the disk can make a redundant root, and that tool is the installer.

Spare disks#

#cloud-config
corium:
  role: controller+worker
  raid:
    - name: data
      level: 1
      devices:
        - /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
        - /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2
      filesystem: ext4
      mountPoint: /var/lib/corium/data

At first boot, the node builds /dev/md/data, formats it, and mounts it. It writes an /etc/fstab entry so that later boots also mount it. It records the array in /etc/mdadm.conf so that the array reassembles.

The configuration reference lists every field.

Use /dev/disk/by-id#

The kernel gives names such as /dev/sdb and /dev/sdc in discovery order. That order can change on the next boot. On a first boot, the order can also be different from the order you expected when you wrote the configuration. To build an array destroys the data on its disks, so the wrong disk name causes serious loss.

A /dev/disk/by-id/... path names the hardware. Use it.

It refuses to destroy data#

The bootstrap stops if a device already holds a filesystem, a partition table, or the metadata of another array:

device /dev/sdb already holds a filesystem or array member signature (TYPE=ext4);
refusing to overwrite it. Set wipe: true on this array to consent to erasing
these devices

wipe: true gives your consent. It is off by default because the two failures do not cost the same. A node that does not finish its bootstrap costs you an afternoon. A disk that Corium erased without a warning cannot be recovered, however careful you are afterwards.

If you run the bootstrap again on a node where the array already exists, the node adopts the array. It does not rebuild the array, and it does not reformat a filesystem that is already on it.

How this differs from runcmd + mdadm#

cloud-init has no RAID support. Its disk_setup module makes partitions and filesystems. For years, its documentation said that mdadm support is planned. So the real alternative to this feature is a runcmd block that calls mdadm by hand. That approach gets three things wrong.

The array exists before k0s starts. This is the most important difference. runcmd runs late, and it races the kubelet. If runcmd loses that race, containerd writes its state into the directory on the root disk. The array mounts over that directory later, and the mount hides the data. The data still uses space on the root disk, but no tool will ever find it. The node then reports a full disk with no clear cause.

Corium builds arrays before it installs k0s. It also writes a drop-in so that the k0s unit waits for the mount:

# /etc/systemd/system/k0scontroller.service.d/10-corium-raid.conf
[Unit]
RequiresMountsFor=/var/lib/corium/data

The fstab entry has nofail. So if the array does not come back, the node still boots and you can log in. Kubernetes refuses to start, with a clear error. It does not run without its storage.

It refuses to destroy data, as described above. A mdadm --create command that you write by hand does not ask.

It is idempotent. If you run the bootstrap twice, it adopts the array and does not rebuild it. It rewrites its own fstab line and does not add a second one.

Checking an array#

cat /proc/mdstat
sudo mdadm --detail /dev/md/data
findmnt /var/lib/corium/data

A healthy mirror shows [UU]. A degraded mirror shows [U_]. While degraded, it continues to serve reads and writes.

Replacing a failed disk#

# 1. Remove the failed member.
sudo mdadm --manage /dev/md/data --remove /dev/disk/by-id/<failed>

# 2. Physically replace it, then add the new one.
sudo mdadm --manage /dev/md/data --add /dev/disk/by-id/<new>

# 3. Watch it rebuild.
cat /proc/mdstat

If you declare a spares: list, steps 1 and 2 happen automatically when a member fails.

The node does not read the corium: block again after the first boot. So this repair needs no configuration change. Also, a later change to the configuration does not undo a manual repair.

Root filesystem on RAID#

Root-on-RAID needs three manual steps that Anaconda does not do for you. If you skip one, the machine does not survive the loss of its first disk. Anaconda alone gives you a mirrored filesystem, not a redundant boot. We tested the procedure below from end to end on a two-disk machine. The test included removing the first disk and booting from the second. Corium does not model this in the corium: block, because the root filesystem is already mounted when the node reads that block.

You make this decision at install time, from the ISO. It is not available on the qcow2 or cloud-image paths, because those ship a fixed disk layout.

Worth weighing first#

A cluster that can lose a node does not need nodes that can lose a disk. When a disk fails, it is usually cheaper to provision the node again. The rest of Corium uses that model, and the image-based upgrade path makes it fast.

Root-on-RAID gives you a root filesystem that survives. At present, it does not reliably give you a boot without interruption.

What is already in place#

The image needs nothing added:

• mdadm ships in fedora-bootc.
• mdadm and its systemd units are in the initramfs. This lets the system assemble an array early enough to mount root from it.
• The Anaconda installer runtime includes mdadm. Its storage module processes part and raid for all payloads that install the OS.

/boot on RAID1 is fine#

The Anaconda GRUB2 backend accepts mdarray for /boot, only at RAID1, with metadata 0.90, 1.0 or 1.2. Members must be partitions, not whole disks. GRUB2.install() then installs to each disk under the array. So you can boot a mirrored /boot from either disk.

Do not put /boot/efi on RAID#

Use one independent, non-RAID ESP per disk. bootupd expects this model. It goes from /boot to the physical disks, and it finds the ESP on each disk. A RAID1 /boot gives it both disks automatically.

error: boot data installation failed: installing component EFI:
Updating EFI firmware variables: Adding new EFI boot entry:
Failed to read /sys/class/block/md125/partition: No such file or directory

Kickstart#

bootc-image-builder embeds a kickstart into an anaconda-iso build. It adds only the ostreecontainer line. You must supply the partitioning, network, locale, and users. You cannot combine a [customizations.installer.kickstart] block with other installer customizations.

Note. This procedure uses the anaconda-iso type. Upstream deprecated that type, and also bootc-image-builder. The kickstart below stays when the ISO moves to its successor. It becomes a file in an installer container, not a build configuration. At that time, we will test it again and rewrite this section. Until then, this is the tested procedure. See ADR 9.

Warning. This kickstart alone is not sufficient. An install from it boots, but it does not survive the loss of its first disk. You must also do three manual steps, which are below. Build a second ESP in %post. Keep all work that is not disk-level out of %post. Add nofail to /boot/efi in /etc/fstab.

# config.toml
[customizations.installer.kickstart]
contents = """
text --non-interactive
zerombr
clearpart --all --initlabel --disklabel=gpt
ignoredisk --only-use=sda,sdb

# One ESP, on the first disk. The second one is built in %post, because
# Anaconda has no kickstart idiom for an unmounted ESP.
part /boot/efi --fstype=efi --size=600 --ondisk=sda

# /boot mirrored, metadata 1.2, members are partitions.
part raid.11 --size=2048 --ondisk=sda --fstype=mdmember
part raid.12 --size=2048 --ondisk=sdb --fstype=mdmember
raid /boot --level=RAID1 --device=boot --fstype=ext4 raid.11 raid.12

# Root mirrored. Sized rather than grown, so space is left at the end of the
# second disk for the ESP that %post creates there.
part raid.21 --size=16384 --ondisk=sda --fstype=mdmember
part raid.22 --size=16384 --ondisk=sdb --fstype=mdmember
raid / --level=RAID1 --device=root --fstype=ext4 raid.21 raid.22

lang en_US.UTF-8
keyboard us
timezone UTC --utc
network --bootproto=dhcp --device=link --activate --onboot=on
reboot
"""

Anaconda requires --fstype=mdmember on the members. Use --device=<name>, not --device=md0, because the value is a name, not a device node. There is no --metadata option. blivet selects the metadata.

The three steps Anaconda leaves to you#

An install with the kickstart above boots correctly. But it does not survive the loss of the first disk. We found each step below when we removed the disk and watched the result.

1. Build a second ESP and register it. Anaconda creates one ESP, on the first disk. It gives this warning during the install:

boot loader stage2 device boot is on a multi-disk array, but boot loader
stage1 device sda1 is not. A drive failure in boot could render the system
unbootable.

The warning is correct. Without a second ESP, the firmware reports No bootable option or device was found and stops.

Kickstart has no idiom for a second, unmounted ESP. Anaconda reads part none --fstype=efi as a mount point with the name none, and the install fails when it binds /mnt/sysimagenone. So size the last RAID member, and do not grow it. This leaves space on the second disk. Then build the ESP in %post:

%post --nochroot
sgdisk --new=3:0:+600M --typecode=3:EF00 /dev/sdb
partprobe /dev/sdb; sleep 2
mkfs.vfat -F32 /dev/sdb3
mkdir -p /tmp/esp2 && mount /dev/sdb3 /tmp/esp2
cp -a /mnt/sysimage/boot/efi/. /tmp/esp2/
sync; umount /tmp/esp2
efibootmgr --create --disk /dev/sdb --part 3 \
  --label "Fedora (mirror)" --loader '\EFI\fedora\shimx64.efi'
%end

2. Do the rest outside %post. On a bootc install, files that %post writes into /mnt/sysimage/etc and /mnt/sysimage/var do not survive. Disk-level work survives. This is why the block above contains only sgdisk, mkfs and efibootmgr. In our test, a sudoers drop-in and a log file written the same way both disappeared without an error.

3. Add nofail to /boot/efi in /etc/fstab. It is easy to miss this step. The machine boots correctly until the first disk fails. Then it goes into emergency mode, although its mirrored filesystems are healthy and mounted. This is the cause:

$ systemctl show boot-efi.mount -p RequiredBy -p WantedBy
RequiredBy=local-fs.target
WantedBy=

fstab mounts /boot/efi by the UUID of the ESP on the first disk. If that disk fails, the mount fails. The mount is RequiredBy and not WantedBy. So local-fs.target also fails, and systemd goes into emergency mode. When you add nofail, the relation changes:

RequiredBy=
WantedBy=local-fs.target

On bootc, /etc is persistent and gets a three-way merge. So this change survives image upgrades.

Verified#

With all three steps in place, we detached the first disk. The machine booted from the second disk alone:

md126 : active raid1 sda2[1]
      2094080 blocks super 1.2 [2/1] [_U]     → /boot
md127 : active raid1 sda1[1]
      16759808 blocks super 1.2 [2/1] [_U]    → /sysroot

$ systemctl is-system-running
degraded
$ findmnt /boot/efi
(not mounted — its disk is gone)

The node started, accepted SSH, and bootc status still answered. The arrays were degraded and continued to serve data. That is the purpose of a mirror.

Rough edges that remain#

• bootc-generic-growpart.service fails on every boot. It finds the partition from /sys/class/block/<name>/partition, and an md device does not have this file: cat: /sys/class/block/md126/partition: No such file or directory. The failure is not fatal. The unit fails and the system reports degraded, but everything else works. The same assumption breaks bootupd when the ESP itself is on RAID ( bootc#947). Set the size of the root array explicitly. Do not expect the unit to grow it.
• After a failure, /boot/efi is not mounted. The surviving ESP is present and the machine boots from it, but fstab still names the failed one. Change fstab to the surviving ESP before the next bootc upgrade. If you do not, bootloader updates have no target.
• We did not test ESP synchronization during an update. bootupd v0.2.28+ updates every ESP it finds ( bootupd#855), and this image includes v0.2.35. But bootupd#1076 is still open. In that issue, secondary ESPs do not receive grub.cfg. After your first upgrade, remove a disk and test again. A test after the install alone is not sufficient.

What does not apply#

People often quote these problems together, as if root-on-RAID cannot work. Read them one at a time:

Fedora CoreOS solves this with a declaration: Ignition’s boot_device.mirror. It builds the array in the initramfs and copies /boot to each disk. bootc has no equivalent. Because of that gap, and not because of this repository, raid[] covers only spare disks.

What RAID is not#

It is not a backup. A mirror copies an rm -rf to both disks as fast as it copies all other writes. It protects against a disk failure, and against nothing else.

It is not cluster redundancy. If you lose a node, you lose its workloads, whether its disks were mirrored or not. If the data is important, keep it in a place that survives the node. For example, use object storage, a replicated volume, or a backup that you restored at least once.

An array is useful for local state that is expensive to rebuild and cheap to keep. Examples are a large image cache, a local persistent volume behind a service that is already replicated, and scratch space that you do not want to lose during a job.

See also#

• Configuration reference §3.11
• Feature support
• mdadm(8)
• Anaconda Kickstart reference