# Upgrades Kubernetes ships with the operating system, so upgrading either means booting a new image. One version axis, one mechanism, one rollback path. Everything on this page was run on a three-controller cluster rather than inferred; the numbers are measured. --- ## The model A Corium node runs an image. Upgrading replaces that image and reboots. There is no in-place package upgrade, no `k0s` binary to swap, and no second mechanism that can move the Kubernetes version independently of the OS. That last point is why [k0s Autopilot is not used](/docs/reference/feature-support/#out-of-scope). Autopilot upgrades Kubernetes by replacing the k0s binary on disk; Corium's lives in the read-only `/usr`. Two mechanisms able to move the version on their own is a worse position than one that cannot. ### What survives | Path | Across an upgrade | |---|---| | `/usr` | Replaced wholesale. It is the image | | `/etc` | Three-way merged: your edits survive, image defaults update around them | | `/var` | Untouched. `/var/lib/k0s`, container storage, the cluster CA, your data | Verified on a live node: after switching image and rebooting, the cluster CA was still in place, the node kept its name, and `corium-bootstrap.service` did not run — the marker in `/var/lib/corium` survived, so the node did not try to bootstrap itself a second time. --- ## Upgrading a node ```bash # See what is running and what, if anything, is staged. sudo bootc status # Pull a newer build of the image this node already tracks. sudo bootc upgrade # Or move it to a different image or tag. sudo bootc switch ghcr.io/corium-os/corium:0.1 ``` From your workstation, `cctl upgrade --image ...` does the same thing plus the drain and the reboot — see [cctl](/docs/reference/cli/). Neither reboots by default. They stage a deployment for the next boot, which is what makes the maintenance window yours to choose: ``` Queued for next boot: ghcr.io/corium-os/corium:0.1 Version: 0.1.0 Digest: sha256:38c2194904b1de400a6f2366760c85dae5e45bb7f5f500612bb428817e850bb6 ``` Add `--apply` to reboot immediately, or reboot when you are ready. ### Only the difference is transferred Measured on a real switch between two builds of this image: ``` layers already present: 65; layers needed: 11 (214.0 MB) Deploying...done (15 seconds) ``` 214 MB moved for an image of roughly 2.3 GB, because the layers the two builds share were already on disk. This is the practical argument for an OS that is an OCI image: upgrades are incremental for the same reason application images are, with no bespoke delta format to maintain. --- ## Upgrading a cluster Nodes are cattle, but the control plane still has a quorum to respect. ### One node at a time With the [management API](/docs/reference/cli/) enabled, one command does the whole sequence for a list of nodes: ```bash cctl upgrade node-1 node-2 node-3 --image ghcr.io/corium-os/corium: ``` It takes them in order and **stops at the first that does not come back on the digest it was sent to** — a rollout that carries on past a broken machine turns one outage into a cluster-wide one — and says how many were done, because a half-upgraded cluster is a decision somebody has to make. Each node is checked fit to lose before it goes down: bootstrapped, k0s running, and the last boot not judged bad by greenboot. Without the API, the same sequence by hand: ```bash # 1. Stop scheduling and move the workloads off. kubectl drain --ignore-daemonsets --delete-emptydir-data # 2. Stage the new image and reboot into it. ssh sudo bootc switch ghcr.io/corium-os/corium: ssh sudo systemctl reboot # 3. Wait for it to come back Ready, then let it take work again. kubectl wait --for=condition=Ready node/ --timeout=5m kubectl uncordon ``` A node that has been drained and rebooted comes back in about 45 seconds and rejoins on its own. It does not re-bootstrap: the marker in `/var/lib/corium` is what stops it. One difference worth knowing: `cctl upgrade` refuses an image the node's own signing policy would accept unsigned, which is what stops a typo rebasing a Kubernetes node onto a desktop image. `bootc switch` by hand does not ask. That check applies to images you build too, and it is the first thing people hit when they derive one: your repository falls under the policy's permissive default, so it is refused until you sign it and say so. See [building your own image](/docs/guides/derived-images/). ### Controllers With three controllers you can lose one and keep etcd quorum, so upgrade them one at a time and wait for `Ready` in between. Never take two out of three down together — that is quorum lost and an API server that stops accepting writes. With five, two at a time is survivable. With two, there is no quorum to lose gracefully and the cluster will be unavailable during each reboot. If you use [HA](/docs/reference/configuration/#38-ha), rebooting the controller holding the virtual IP moves it to another controller. Clients pointed at the VIP reconnect on their own; the failover was verified by hard-stopping the holder. --- ## A node that breaks rolls itself back Staging an upgrade and rebooting is only safe if something notices when the new image does not work. Nodes run [greenboot](https://github.com/fedora-iot/greenboot) health checks at boot; if a check fails and there is a previous deployment to return to, the node rolls back on its own. Corium ships one check: is k0s running and answering? ``` corium: k0scontroller.service is running and answering greenboot health-check passed. Set grubenv: boot_success=1 ``` **What it deliberately does not check is the interesting part.** It does not require the node to be `Ready` in Kubernetes. A node can be legitimately NotReady for reasons that have nothing to do with the image — no CNI installed yet, a control plane still coming back, a cluster-wide problem — and rolling back the OS would fix none of them while taking a healthy machine out of service at the worst possible moment. A health check that is too strict is worse than none. The check also allows five minutes for k0s to start, since unpacking its supervised binaries and bringing up etcd is not instant, and passes trivially on a node that was never bootstrapped. A node only rolls back if an upgrade actually staged something. greenboot records the deployment it expects to boot, and the rollback path is gated on that record existing — so a health check that fails on a node nobody upgraded gets you a warning and manual intervention, not a surprise trip to an older image. Verified end to end, by pushing an image whose health check always fails and letting the node take it: ``` boot 1 check fails First health check failure, setting boot counter to 3 boot 2 check fails Boot counter is 2, rebooting to try again boot 3 check fails Boot counter is 1, rebooting to try again boot 4 check fails Boot counter exhausted ... initiating rollback Rollback successful boot 5 healthy greenboot health-check passed. Set grubenv: boot_success=1 ``` Four attempts on the broken image, then back to the previous one, which came up and stayed up. `GREENBOOT_MAX_BOOT_ATTEMPTS` in `/etc/greenboot/greenboot.conf` sets the count. Once rolled back, the image that failed is the one that gets replaced, so the node does not oscillate. Add your own checks by dropping executables in `/etc/greenboot/check/required.d/`. A non-zero exit fails the boot. ## Rolling back ```bash sudo bootc rollback sudo systemctl reboot ``` Or remotely, if the node still answers: ```bash cctl rollback # marks the previous image; does not reboot ``` `cctl rollback` deliberately stops there. It exists because somebody is already having a bad day, and taking the node down at a moment they did not choose would not help — the reboot stays yours to schedule. Rollback is instant and downloads nothing — `Next boot: rollback deployment`. It reorders boot entries between the deployment you are running and the previous one, both of which are already on disk. It is also symmetric: rolling back makes the image you left the new rollback target, so running it again returns you to where you were. Verified in both directions on a live controller. ```bash sudo bootc status # shows booted, staged and rollback ``` Two things rollback does **not** undo: - **Anything written to `/var`.** A Kubernetes version that migrated etcd data does not un-migrate because you booted an older image. Read the k0s release notes before skipping Kubernetes minor versions. - **Cluster state.** Rolling back one node does not roll back the cluster. --- ## Choosing what to track A release publishes a ladder of tags, so a node can choose how much movement it accepts. | Tag | Moves | You get | |---|---|---| | `corium@sha256:...` | Never | Exactly one image. The strongest pin | | `corium:0.1.0` | Never | One release | | `corium:0.1` | On patch releases | Fixes, no new behaviour | | `corium:latest` | On every release | Whatever is newest | | `corium:main` | On every push to `main` | Development builds, unreleased | **Track `0.1`.** It moves only on patch releases, which do not change behaviour, so it picks up fixes without you deciding anything. Pin `0.1.0` exactly if you would rather choose the moment yourself. Read the changelog before moving to `0.2`: a minor release may require a configuration change, and it will say which. **Below 1.0 there is no major rung.** Semantic versioning reserves `0.y.z` for initial development and lets a minor release break things, so a `0` tag meaning "new features, no breaking changes" would promise exactly what the version number withholds. It is not published. When Corium reaches 1.0, a `1` rung joins the ladder with that meaning, and it will be true: | Tag | Moves | You get | |---|---|---| | `corium:1.4` | On patch releases | Fixes, no new behaviour | | `corium:1` | On minor releases | New features, no breaking changes | `latest` is a claim about recency, not about compatibility. It crosses minor releases below 1.0 and major ones above, which is where breaking changes live by definition. It exists so that `podman pull ghcr.io/corium-os/corium` returns something; it is not a tag to run a cluster on. Prereleases publish only their exact tag: `0.2.0-rc.1` never becomes `0.2` or `latest`, so a node following a stable tag will not pick up a release candidate. `bootc status` always reports the digest actually booted, whatever the tag said at the time — which is what you want in an incident. ### Unattended upgrades A node does nothing on its own by default. A Kubernetes node that reboots unprompted is an outage nobody scheduled, so `bootc-fetch-apply-updates.timer` is masked in the image. Two levels are available when you want more: ```yaml corium: upgrades: automatic: download # none (default) | download | apply schedule: "Mon *-*-* 03:00:00" # systemd OnCalendar, default daily ``` | Policy | What the node does | Reboots itself | |---|---|---| | `none` | Nothing. The default | No | | `download` | Stages a newer image and queues it for the next boot, whenever that is | **No** | | `apply` | Stages it and reboots | **Yes** | **`download` is the one most clusters want.** The fetch and the deployment happen unattended, so the reboot you schedule becomes near-instant — the expensive part is already done. You still choose the moment, drain first, and go one node at a time. `apply` drains the node before rebooting: it cordons, evicts the pods, reboots into the staged image, and uncordons once k0s is back. A node upgrading itself therefore reschedules its workloads rather than killing them. Two behaviours worth knowing, because both are deliberate: - **A drain that cannot finish cancels the upgrade.** A pod disruption budget refusing an eviction is the system working, not a fault. The node uncordons itself, stays on its current image with the new one still staged, and tries again at the next tick. Forcing a reboot past a PDB would defeat the point of having one. - **A plain worker reboots undrained.** Draining needs cluster admin credentials, and only a node running a control plane has them locally. On `worker` nodes, `apply` behaves as it did before. Use `download` there and drive the reboot from somewhere that can talk to the API. - **The new image stays locked until the drain has succeeded.** `apply` stages with `bootc upgrade --download-only`, which leaves the deployment *locked for finalization*, and unlocks it with `--from-downloaded` only once the node is drained. A reboot for any other reason in between — an operator, a crash, a power cut halfway through the drain — comes back up on the image the node was already running, rather than half-applying an upgrade nobody scheduled. `schedule` takes any [systemd OnCalendar](https://www.freedesktop.org/software/systemd/man/systemd.time.html) expression. A randomised delay of up to an hour is applied on top, so a fleet does not arrive at the registry in lockstep, and missed checks are caught up after a node has been off rather than waiting for the next window. Check what a node has staged: ```bash sudo bootc status systemctl list-timers 'corium-upgrade-*' 'bootc-*' ``` --- ## Verifying what you are about to boot Every image Corium publishes -- from a release tag and from `main` alike -- is signed with [cosign](https://docs.sigstore.dev/), keyless: there is no private key, and the signing identity is the GitHub Actions workflow itself. The signature is attached to the image digest rather than to a tag, because tags move and a signature on a moving tag says nothing about what it points at now. ```bash cosign verify ghcr.io/corium-os/corium:0.1 \ --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` Verified output looks like this: ``` Verification for ghcr.io/corium-os/corium:0.1 -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates ``` If `cosign verify` hangs with no output at all, it is probably your registry credential helper waiting on something — a locked macOS keychain does this. Running it with an empty `DOCKER_CONFIG` is a quick way to tell: ```bash mkdir -p /tmp/emptycfg && echo '{}' > /tmp/emptycfg/config.json DOCKER_CONFIG=/tmp/emptycfg cosign verify ghcr.io/corium-os/corium:0.1 \ --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` ### Nodes enforce it Images are signed **twice**, because the two signatures answer different questions. | Signature | Proves | Used by | |---|---|---| | Keyless | Which workflow run built this image, recorded in a public transparency log | A human, running `cosign verify` | | Key | That the image came from this project | The node, at pull time | A Corium node ships a policy requiring the key signature for `ghcr.io/corium-os/corium`, with the public key at `/usr/share/corium/cosign.pub`. An image from that repository that is not signed is refused: ``` Source image rejected: A signature was required, but no signature exists ``` The policy is deliberately narrow. Everything else stays permissive, because k0s pulls its own images from quay.io and docker.io — a blanket policy would break the cluster rather than secure it. Verify it against a node yourself: ```bash cosign verify --key /usr/share/corium/cosign.pub ghcr.io/corium-os/corium:0.1 ``` **Why two signatures rather than one.** A node cannot enforce the keyless one: `containers-policy.json` matches a signer by `subjectEmail`, and a GitHub Actions certificate carries a URI instead, so enforcement fails with `Required email ... not found (got [])`. That is a limitation of `containers-image`, not a configuration mistake — worth knowing before you spend an afternoon on it. See [bootc: image signatures](https://bootc.dev/bootc/security.html). --- ## Upgrading Kubernetes specifically The Kubernetes version is whatever `build/k0s.lock` pins, so upgrading it means changing that file, rebuilding, and rolling the new image out as above. ``` K0S_VERSION=v1.36.4+k0s.0 K0S_SHA256_amd64=... ``` `mise run k0s-lock vX.Y.Z+k0s.N` rewrites it: it downloads both architectures' binaries and hashes them, rather than trusting a checksum published beside the download. Review the resulting diff — it is what every image build verifies against from then on. CI checks that the image really ships the version the lock file names, so a lock file and an image cannot silently disagree. Before skipping a Kubernetes minor version, read the [k0s release notes](https://docs.k0sproject.io/stable/releases/): k0s follows upstream Kubernetes, which does not support skipping minors on the control plane.