Upgrades

Kubernetes ships with the operating system, so upgrading either means booting a new image. One version axis, one mechanism, one rollback path.

Everything on this page was run on a three-controller cluster rather than inferred; the numbers are measured.

The model#

A Corium node runs an image. Upgrading replaces that image and reboots. There is no in-place package upgrade, no k0s binary to swap, and no second mechanism that can move the Kubernetes version independently of the OS.

That last point is why k0s Autopilot is not used. Autopilot upgrades Kubernetes by replacing the k0s binary on disk; Corium’s lives in the read-only /usr. Two mechanisms able to move the version on their own is a worse position than one that cannot.

What survives#

Verified on a live node: after switching image and rebooting, the cluster CA was still in place, the node kept its name, and corium-bootstrap.service did not run — the marker in /var/lib/corium survived, so the node did not try to bootstrap itself a second time.

Upgrading a node#

# See what is running and what, if anything, is staged.
sudo bootc status

# Pull a newer build of the image this node already tracks.
sudo bootc upgrade

# Or move it to a different image or tag.
sudo bootc switch ghcr.io/corium-os/corium:0.1

From your workstation, cctl upgrade <node> --image ... does the same thing plus the drain and the reboot — see cctl.

Neither reboots by default. They stage a deployment for the next boot, which is what makes the maintenance window yours to choose:

Queued for next boot: ghcr.io/corium-os/corium:0.1
  Version: 0.1.0
  Digest: sha256:38c2194904b1de400a6f2366760c85dae5e45bb7f5f500612bb428817e850bb6

Add --apply to reboot immediately, or reboot when you are ready.

Only the difference is transferred#

Measured on a real switch between two builds of this image:

layers already present: 65; layers needed: 11 (214.0 MB)
Deploying...done (15 seconds)

214 MB moved for an image of roughly 2.3 GB, because the layers the two builds share were already on disk. This is the practical argument for an OS that is an OCI image: upgrades are incremental for the same reason application images are, with no bespoke delta format to maintain.

Upgrading a cluster#

Nodes are cattle, but the control plane still has a quorum to respect.

One node at a time#

With the management API enabled, one command does the whole sequence for a list of nodes:

cctl upgrade node-1 node-2 node-3 --image ghcr.io/corium-os/corium:<tag>

It takes them in order and stops at the first that does not come back on the digest it was sent to — a rollout that carries on past a broken machine turns one outage into a cluster-wide one — and says how many were done, because a half-upgraded cluster is a decision somebody has to make. Each node is checked fit to lose before it goes down: bootstrapped, k0s running, and the last boot not judged bad by greenboot.

Without the API, the same sequence by hand:

# 1. Stop scheduling and move the workloads off.
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# 2. Stage the new image and reboot into it.
ssh <node> sudo bootc switch ghcr.io/corium-os/corium:<tag>
ssh <node> sudo systemctl reboot

# 3. Wait for it to come back Ready, then let it take work again.
kubectl wait --for=condition=Ready node/<node> --timeout=5m
kubectl uncordon <node>

A node that has been drained and rebooted comes back in about 45 seconds and rejoins on its own. It does not re-bootstrap: the marker in /var/lib/corium is what stops it.

One difference worth knowing: cctl upgrade refuses an image the node’s own signing policy would accept unsigned, which is what stops a typo rebasing a Kubernetes node onto a desktop image. bootc switch by hand does not ask.

That check applies to images you build too, and it is the first thing people hit when they derive one: your repository falls under the policy’s permissive default, so it is refused until you sign it and say so. See building your own image.

Controllers#

With three controllers you can lose one and keep etcd quorum, so upgrade them one at a time and wait for Ready in between. Never take two out of three down together — that is quorum lost and an API server that stops accepting writes.

With five, two at a time is survivable. With two, there is no quorum to lose gracefully and the cluster will be unavailable during each reboot.

If you use HA, rebooting the controller holding the virtual IP moves it to another controller. Clients pointed at the VIP reconnect on their own; the failover was verified by hard-stopping the holder.

A node that breaks rolls itself back#

Staging an upgrade and rebooting is only safe if something notices when the new image does not work. Nodes run greenboot health checks at boot; if a check fails and there is a previous deployment to return to, the node rolls back on its own.

Corium ships one check: is k0s running and answering?

corium: k0scontroller.service is running and answering
greenboot health-check passed.
Set grubenv: boot_success=1

What it deliberately does not check is the interesting part. It does not require the node to be Ready in Kubernetes. A node can be legitimately NotReady for reasons that have nothing to do with the image — no CNI installed yet, a control plane still coming back, a cluster-wide problem — and rolling back the OS would fix none of them while taking a healthy machine out of service at the worst possible moment. A health check that is too strict is worse than none.

The check also allows five minutes for k0s to start, since unpacking its supervised binaries and bringing up etcd is not instant, and passes trivially on a node that was never bootstrapped.

A node only rolls back if an upgrade actually staged something. greenboot records the deployment it expects to boot, and the rollback path is gated on that record existing — so a health check that fails on a node nobody upgraded gets you a warning and manual intervention, not a surprise trip to an older image.

Verified end to end, by pushing an image whose health check always fails and letting the node take it:

boot 1   check fails   First health check failure, setting boot counter to 3
boot 2   check fails   Boot counter is 2, rebooting to try again
boot 3   check fails   Boot counter is 1, rebooting to try again
boot 4   check fails   Boot counter exhausted ... initiating rollback
                       Rollback successful
boot 5   healthy       greenboot health-check passed.  Set grubenv: boot_success=1

Four attempts on the broken image, then back to the previous one, which came up and stayed up. GREENBOOT_MAX_BOOT_ATTEMPTS in /etc/greenboot/greenboot.conf sets the count. Once rolled back, the image that failed is the one that gets replaced, so the node does not oscillate.

Add your own checks by dropping executables in /etc/greenboot/check/required.d/. A non-zero exit fails the boot.

Rolling back#

sudo bootc rollback
sudo systemctl reboot

Or remotely, if the node still answers:

cctl rollback <node>     # marks the previous image; does not reboot

cctl rollback deliberately stops there. It exists because somebody is already having a bad day, and taking the node down at a moment they did not choose would not help — the reboot stays yours to schedule.

Rollback is instant and downloads nothing — Next boot: rollback deployment. It reorders boot entries between the deployment you are running and the previous one, both of which are already on disk.

It is also symmetric: rolling back makes the image you left the new rollback target, so running it again returns you to where you were. Verified in both directions on a live controller.

sudo bootc status   # shows booted, staged and rollback

Two things rollback does not undo:

• Anything written to /var. A Kubernetes version that migrated etcd data does not un-migrate because you booted an older image. Read the k0s release notes before skipping Kubernetes minor versions.
• Cluster state. Rolling back one node does not roll back the cluster.

Choosing what to track#

A release publishes a ladder of tags, so a node can choose how much movement it accepts.

Track 0.1. It moves only on patch releases, which do not change behaviour, so it picks up fixes without you deciding anything. Pin 0.1.0 exactly if you would rather choose the moment yourself. Read the changelog before moving to 0.2: a minor release may require a configuration change, and it will say which.

Below 1.0 there is no major rung. Semantic versioning reserves 0.y.z for initial development and lets a minor release break things, so a 0 tag meaning “new features, no breaking changes” would promise exactly what the version number withholds. It is not published. When Corium reaches 1.0, a 1 rung joins the ladder with that meaning, and it will be true:

latest is a claim about recency, not about compatibility. It crosses minor releases below 1.0 and major ones above, which is where breaking changes live by definition. It exists so that podman pull ghcr.io/corium-os/corium returns something; it is not a tag to run a cluster on.

Prereleases publish only their exact tag: 0.2.0-rc.1 never becomes 0.2 or latest, so a node following a stable tag will not pick up a release candidate.

bootc status always reports the digest actually booted, whatever the tag said at the time — which is what you want in an incident.

Unattended upgrades#

A node does nothing on its own by default. A Kubernetes node that reboots unprompted is an outage nobody scheduled, so bootc-fetch-apply-updates.timer is masked in the image.

Two levels are available when you want more:

corium:
  upgrades:
    automatic: download      # none (default) | download | apply
    schedule: "Mon *-*-* 03:00:00"   # systemd OnCalendar, default daily

download is the one most clusters want. The fetch and the deployment happen unattended, so the reboot you schedule becomes near-instant — the expensive part is already done. You still choose the moment, drain first, and go one node at a time.

apply drains the node before rebooting: it cordons, evicts the pods, reboots into the staged image, and uncordons once k0s is back. A node upgrading itself therefore reschedules its workloads rather than killing them.

Two behaviours worth knowing, because both are deliberate:

• A drain that cannot finish cancels the upgrade. A pod disruption budget refusing an eviction is the system working, not a fault. The node uncordons itself, stays on its current image with the new one still staged, and tries again at the next tick. Forcing a reboot past a PDB would defeat the point of having one.
• A plain worker reboots undrained. Draining needs cluster admin credentials, and only a node running a control plane has them locally. On worker nodes, apply behaves as it did before. Use download there and drive the reboot from somewhere that can talk to the API.
• The new image stays locked until the drain has succeeded. apply stages with bootc upgrade --download-only, which leaves the deployment locked for finalization, and unlocks it with --from-downloaded only once the node is drained. A reboot for any other reason in between — an operator, a crash, a power cut halfway through the drain — comes back up on the image the node was already running, rather than half-applying an upgrade nobody scheduled.

schedule takes any systemd OnCalendar expression. A randomised delay of up to an hour is applied on top, so a fleet does not arrive at the registry in lockstep, and missed checks are caught up after a node has been off rather than waiting for the next window.

Check what a node has staged:

sudo bootc status
systemctl list-timers 'corium-upgrade-*' 'bootc-*'

Verifying what you are about to boot#

Every image Corium publishes – from a release tag and from main alike – is signed with cosign, keyless: there is no private key, and the signing identity is the GitHub Actions workflow itself. The signature is attached to the image digest rather than to a tag, because tags move and a signature on a moving tag says nothing about what it points at now.

cosign verify ghcr.io/corium-os/corium:0.1 \
  --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Verified output looks like this:

Verification for ghcr.io/corium-os/corium:0.1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate
    authority certificates

If cosign verify hangs with no output at all, it is probably your registry credential helper waiting on something — a locked macOS keychain does this. Running it with an empty DOCKER_CONFIG is a quick way to tell:

mkdir -p /tmp/emptycfg && echo '{}' > /tmp/emptycfg/config.json
DOCKER_CONFIG=/tmp/emptycfg cosign verify ghcr.io/corium-os/corium:0.1 \
  --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Nodes enforce it#

Images are signed twice, because the two signatures answer different questions.

A Corium node ships a policy requiring the key signature for ghcr.io/corium-os/corium, with the public key at /usr/share/corium/cosign.pub. An image from that repository that is not signed is refused:

Source image rejected: A signature was required, but no signature exists

The policy is deliberately narrow. Everything else stays permissive, because k0s pulls its own images from quay.io and docker.io — a blanket policy would break the cluster rather than secure it.

Verify it against a node yourself:

cosign verify --key /usr/share/corium/cosign.pub ghcr.io/corium-os/corium:0.1

Why two signatures rather than one. A node cannot enforce the keyless one: containers-policy.json matches a signer by subjectEmail, and a GitHub Actions certificate carries a URI instead, so enforcement fails with Required email ... not found (got []). That is a limitation of containers-image, not a configuration mistake — worth knowing before you spend an afternoon on it.

See bootc: image signatures.

Upgrading Kubernetes specifically#

The Kubernetes version is whatever build/k0s.lock pins, so upgrading it means changing that file, rebuilding, and rolling the new image out as above.

K0S_VERSION=v1.36.4+k0s.0
K0S_SHA256_amd64=...

mise run k0s-lock vX.Y.Z+k0s.N rewrites it: it downloads both architectures' binaries and hashes them, rather than trusting a checksum published beside the download. Review the resulting diff — it is what every image build verifies against from then on.

CI checks that the image really ships the version the lock file names, so a lock file and an image cannot silently disagree.

Before skipping a Kubernetes minor version, read the k0s release notes: k0s follows upstream Kubernetes, which does not support skipping minors on the control plane.