Upgrades

The operating system and Kubernetes upgrade on separate axes. To move the OS, you boot a new image. This has one mechanism and one rollback path. To move Kubernetes, you use kubernetes.version or cctl k8s upgrade, and you need no new image. Upgrading Kubernetes specifically below covers it. Only one operation is not symmetric: to roll a Kubernetes version back. k0s does not support a downgrade of a minor.

This page tells you how to:

• Move one node to a new image.
• Roll a whole cluster, one node at a time.
• Roll back when an image is wrong.
• Choose the tag that a node follows.
• Verify the image that you are about to boot.

It applies to every Corium node, for every install path: ISO, qcow2, cloud image, or an image that you derived yourself.

The model#

A Corium node runs an image. An upgrade replaces that image and reboots. There is no in-place package upgrade, no k0s binary to swap, and no second mechanism that can move the Kubernetes version independently of the OS.

For this last reason, Corium does not use k0s Autopilot. Autopilot upgrades Kubernetes by replacing the k0s binary on disk. The Corium binary is in the read-only /usr. Two mechanisms that can each change the version on their own are worse than one that cannot.

What survives#

This was tested on a live node. After the image switch and the reboot, the cluster CA was still in place, and the node kept its name. corium-bootstrap.service did not run. The marker in /var/lib/corium stayed, so the node did not try to bootstrap itself a second time.

All results below come from measurements on a three-controller cluster, not from assumptions.

Upgrading a node#

Before you start#

• A node on which you can run sudo, locally or over SSH.
• An image tag. See choosing what to track.
• For the cctl path only: the management API enabled on that node, and a client certificate for it.

# See what is running and what, if anything, is staged.
sudo bootc status

# Pull a newer build of the image this node already tracks.
sudo bootc upgrade

# Or move it to a different image or tag.
sudo bootc switch ghcr.io/corium-os/corium:0.5

From your workstation, cctl upgrade <node> --image ... does the same thing. It also drains and reboots the node. See cctl.

By default, neither command reboots. They stage a deployment for the next boot. Thus you choose the maintenance window:

Queued for next boot: ghcr.io/corium-os/corium:0.5
  Version: 0.5.0
  Digest: sha256:38c2194904b1de400a6f2366760c85dae5e45bb7f5f500612bb428817e850bb6

To reboot immediately, add --apply. Or reboot when you are ready.

Only the difference is transferred#

This output comes from a real switch between two builds of this image:

layers already present: 65; layers needed: 11 (214.0 MB)
Deploying...done (15 seconds)

The switch moved 214 MB for an image of approximately 2.3 GB. The layers that the two builds share were already on disk. This is the practical argument for an OS that is an OCI image. Upgrades are incremental for the same reason that application images are, and there is no custom delta format to maintain.

Upgrading a cluster#

Nodes are replaceable, but the control plane still has a quorum that you must keep.

One node at a time#

If the management API is enabled, one command does the whole sequence for a list of nodes:

Warning. cctl upgrade refuses an image that the signing policy of the node accepts without a signature. This stops a typo from rebasing a Kubernetes node onto a desktop image. It is also the first problem that you meet when you derive your own image. Your repository falls under the permissive default of the policy. Thus cctl upgrade refuses it until you sign it and declare it in the policy. See building your own image. If you run bootc switch by hand, it does not do this check.

cctl upgrade node-1 node-2 node-3 --image ghcr.io/corium-os/corium:<tag>

Before it changes any node, the command makes sure that each node meets these conditions:

• The node is reachable.
• The node is safe to lose: it is bootstrapped, k0s runs, and greenboot did not mark the last boot as bad.
• The node is in the same cluster as the others.
• The node is listed only once.
• The node accepts the image under its own signing policy.

If one node fails a check, the command stops and changes nothing. Thus you find a wrong address or a refused image before the first nodes reboot. To run only that check, add --dry-run.

Then the command takes the nodes in order, and asks each node again if it is safe to lose. It stops at the first node that does not come back on the digest it was sent to, and tells you how many nodes it upgraded.

Both of these behaviors are deliberate. A rollout that continues past a broken machine turns one outage into an outage of the whole cluster. Also, a half-upgraded cluster needs a decision from a person. It is not a number to hide.

Without the API, do the same sequence by hand:

# 1. Stop scheduling and move the workloads off.
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# 2. Stage the new image and reboot into it.
ssh <node> sudo bootc switch ghcr.io/corium-os/corium:<tag>
ssh <node> sudo systemctl reboot

# 3. Wait for it to come back Ready, then let it take work again.
kubectl wait --for=condition=Ready node/<node> --timeout=5m
kubectl uncordon <node>

After the drain and the reboot, the node comes back in approximately 45 seconds and rejoins on its own. It does not bootstrap again, because the marker in /var/lib/corium stops it.

Controllers#

With three controllers, you can lose one and keep etcd quorum. Thus upgrade them one at a time, and wait for Ready between each one. Never stop two of the three controllers at the same time. If you do, the cluster loses quorum, and the API server stops accepting writes.

cctl upgrade and cctl k8s upgrade enforce this rule. Before and after each controller, they make sure that every etcd member is healthy. If a member is not healthy, they stop. See etcd around controllers.

With five controllers, the cluster survives two at a time. With two controllers, the cluster has no quorum to spare, and it is unavailable during each reboot.

If you use HA, a reboot of the controller that holds the virtual IP moves the virtual IP to another controller. Clients that point at the VIP reconnect on their own. This failover was tested with a hard stop of the controller that held the VIP.

A node that breaks rolls itself back#

Nodes run greenboot health checks at boot. If a check fails and there is a previous deployment to return to, the node rolls back on its own. Thus you can stage an image and reboot without supervision.

This table comes from a test with an image whose health check always fails:

GREENBOOT_MAX_BOOT_ATTEMPTS in /etc/greenboot/greenboot.conf sets the count. After the rollback, the next upgrade replaces the image that failed. Thus the node does not switch back and forth between the two images. A rollback also occurs only if an upgrade staged something. If a check fails on a node that nobody upgraded, you get a warning and must act manually. The node does not move to an older image without warning.

To add your own checks, put executables in /etc/greenboot/check/required.d/. A non-zero exit fails the boot.

Corium ships one check: is k0s running and answering? It does not require the node to be Ready in Kubernetes, on purpose. A node can be NotReady for reasons that the image cannot fix, for example:

• No CNI is installed yet.
• A control plane is still coming back.
• The whole cluster has a problem.

An OS rollback fixes none of these. It only takes a healthy machine out of service at the worst possible moment. A health check that is too strict is worse than no check. The check gives k0s five minutes to start, because k0s needs time to unpack its supervised binaries and start etcd. On a node that was never bootstrapped, the check always passes.

Rolling back#

sudo bootc rollback
sudo systemctl reboot

Or remotely, if the node still answers:

cctl rollback <node>     # marks the previous image; does not reboot

cctl rollback stops there on purpose. When you need it, you already have a problem. A reboot of the node at a moment that you did not choose does not help. You schedule the reboot.

A rollback is instant and downloads nothing: Next boot: rollback deployment. It changes the order of the boot entries for the deployment that you run and the previous deployment. Both are already on disk.

A rollback is also symmetric. The image that you left becomes the new rollback target. If you run the rollback again, you return to where you were. This was tested in both directions on a live controller.

sudo bootc status   # shows booted, staged and rollback

Two things rollback does not undo:

• Anything written to /var. If a Kubernetes version migrated etcd data, an older image does not reverse the migration. Before you skip Kubernetes minor versions, read the k0s release notes.
• Cluster state. Rolling back one node does not roll back the cluster.

Choosing what to track#

Each release publishes a ladder of tags. Thus each node can choose how much change it accepts.

Track 0.5. It moves only on patch releases, which do not change behavior. Thus the node gets fixes without a decision from you. If you prefer to choose the moment yourself, pin 0.5.0 exactly. Before you move to 0.6, read the changelog. A minor release can require a configuration change, and the changelog names it.

Moving from 0.4 to 0.5 changes a node’s posture. From 0.5.0, the management API is on by default. A node that names no api: block serves its enrolment route during the minute between boot and bootstrap. The pairing code on its console protects that route. To keep the old behavior, set api.enabled: false. For this reason, the change is a minor release and not a patch. Nothing on the 0.4 rung moves, so a node gets it only when you tell it to. See ADR 11.

The ladder does not offer three things:

• There is no major rung below 1.0. Semantic versioning lets a 0.y minor release break things. A 0 tag promises a stability that the version number does not give.
• latest tells you which release is newest. It does not tell you about compatibility. It crosses the minor releases, which is where breaking changes occur. Do not run a cluster on it.
• Prereleases publish only their exact tag. 0.5.0-rc.1 never becomes 0.5 or latest. Thus a node that follows a stable tag does not get a release candidate.

bootc status always reports the digest that the node booted, whatever the tag pointed to at that time. During an incident, this is the information that you need.

Unattended upgrades#

By default, a node does nothing on its own. A Kubernetes node that reboots without a request causes an outage that nobody scheduled. Thus the image masks bootc-fetch-apply-updates.timer.

If you want more, two levels are available:

corium:
  upgrades:
    automatic: download      # none (default) | download | apply
    schedule: "Mon *-*-* 03:00:00"   # systemd OnCalendar, default daily

download is the one most clusters want. The fetch and the deployment occur without supervision. Thus the reboot that you schedule is almost instant, because the slow part is already done. You still choose the moment, drain first, and do one node at a time.

apply drains the node before it reboots. It cordons the node, evicts the pods, reboots into the staged image, and uncordons the node when k0s is back. Thus a node that upgrades itself moves its workloads to other nodes and does not kill them.

These behaviors are deliberate:

• A drain that cannot finish cancels the upgrade. When a pod disruption budget refuses an eviction, the system works correctly. It is not a fault. The node uncordons itself and stays on its current image. The new image stays staged, and the node tries again at the next scheduled run. A reboot that ignores a PDB defeats the purpose of the PDB.
• A plain worker reboots without a drain. A drain needs cluster admin credentials, and only a node that runs a control plane has them locally. On worker nodes, apply reboots without a drain. On those nodes, use download, and start the reboot from a machine that can talk to the API.
• A single-node cluster also reboots without a drain. A drain of the only node has nowhere to move its pods. Eviction then blocks on their disruption budgets, and the drain never finishes. Under the rule above, this cancels every upgrade. Thus a single node (or any cluster of one) skips the drain and reboots directly into the staged image. Its pods come back when the node does. This lets cctl upgrade reboot a one-node cluster. Otherwise, the image stays staged forever.
• The new image stays locked until the drain succeeds. apply stages with bootc upgrade --download-only, which leaves the deployment locked for finalization. It unlocks the deployment with --from-downloaded only after the node is drained. A reboot for a different reason in between can come from an operator, a crash, or a power cut during the drain. After such a reboot, the node comes back on the image that it already ran. It does not half-apply an upgrade that nobody scheduled.

schedule takes any systemd OnCalendar expression. Corium adds a random delay of up to one hour, so that all nodes do not contact the registry at the same time. If a node was off, it runs the missed checks when it starts again. It does not wait for the next window.

To see what a node staged, run:

sudo bootc status
systemctl list-timers 'corium-upgrade-*' 'bootc-*'

Verifying what you are about to boot#

Corium signs every image that it publishes, from a release tag and from main. It uses cosign in keyless mode. There is no private key, and the signing identity is the GitHub Actions workflow itself. The signature is attached to the image digest and not to a tag. Tags move, and a signature on a moving tag tells you nothing about the image that the tag points to now.

cosign verify ghcr.io/corium-os/corium:0.5 \
  --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

A successful verification prints this output:

Verification for ghcr.io/corium-os/corium:0.5 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate
    authority certificates

If cosign verify stops with no output at all, your registry credential helper is probably waiting for something. A locked macOS keychain causes this. To find out, run the command with an empty DOCKER_CONFIG:

mkdir -p /tmp/emptycfg && echo '{}' > /tmp/emptycfg/config.json
DOCKER_CONFIG=/tmp/emptycfg cosign verify ghcr.io/corium-os/corium:0.5 \
  --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Nodes enforce it#

Corium signs images twice, because the two signatures answer different questions.

A Corium node ships a policy that requires the key signature for ghcr.io/corium-os/corium. The public key is at /usr/share/corium/cosign.pub. The node refuses an image from that repository that is not signed:

Source image rejected: A signature was required, but no signature exists

The policy is narrow on purpose. Everything else stays permissive, because k0s pulls its own images from quay.io and docker.io. A policy for all registries breaks the cluster and does not make it more secure.

To verify the image yourself with the key from a node, run:

cosign verify --key /usr/share/corium/cosign.pub ghcr.io/corium-os/corium:0.5

Why two signatures rather than one. A node cannot enforce the keyless signature. containers-policy.json matches a signer by subjectEmail, but a GitHub Actions certificate carries a URI instead. Thus enforcement fails with Required email ... not found (got []). This is a limitation of containers-image, not a configuration mistake. Know this before you spend an afternoon on it.

See bootc: image signatures.

Upgrading Kubernetes specifically#

Most of the time you do not need this section. With ADR 10, the Kubernetes version is a separate axis. Set kubernetes.version, or roll a cluster with cctl k8s upgrade. No OS image changes. See reference §3.20.

A controller snapshots its datastore first#

Before a controller stops k0s to change its version, it runs k0s backup. This happens after the drain, while the old version still runs, because k0s backup needs k0s to be up. The archive holds the datastore, the PKI and k0s.yaml. The datastore is etcd on controller and controller+worker, and kine/SQLite on single. A worker has no datastore, so it takes no snapshot.

The API returns the path, and cctl k8s upgrade and cctl apply print it. If k0s does not come back, the error names the path too. Restore from that archive. See restore is manual.

If the snapshot fails, the node cancels the change. Nothing is stopped yet. So the node uncordons itself and stays on its version. The usual causes are a full disk, or k0s not running. Fix the cause, then try again. The snapshot is part of the change that continues after the drain starts. So it finishes even if your client disconnects.

This section covers the other axis: how to change the version that an image ships, and the versions for which it accepts extensions. Both are in build/k0s.lock:

K0S_FLOOR=v1.36.5+k0s.0

# version         arch    sha256
v1.34.12+k0s.0    amd64   8536acf9...
v1.35.9+k0s.0     amd64   517b88e1...
v1.36.5+k0s.0     amd64   377cdb37...

K0S_FLOOR is baked into /usr/bin/k0s. A node runs this version when its configuration names no version. The table is the window: the versions for which this image builds and accepts system extensions. The window is three minors wide. To change the floor, rebuild the image and roll it out as described above. To add a version to the window, merge the change. Then the extension is published, and any node can ask for it without a new image.

mise run k0s-window vX.Y.Z+k0s.N adds a version. It downloads the binaries for both architectures and hashes them. It does not trust a checksum published next to the download. Review the resulting diff, because every later build verifies against it.

CI makes sure that the image ships the floor that the lock file names. Thus a lock file and an image cannot disagree without a failure.

Before you skip a Kubernetes minor version, read the k0s release notes. k0s follows upstream Kubernetes, which does not support skipping minors on the control plane. A node refuses such a change immediately and does not attempt it.