ADR 10 — Kubernetes version axis

Status: accepted

Amends decisions 3, 4 and 5, and adds decision 16.

Context#

Corium ships one Kubernetes version per OS image. build/k0s.lock pins a release and two checksums. build/scripts/install-k0s.sh fetches that release into /usr/bin/k0s at build time. Decision 3 states the rule that makes this deliberate, not accidental: the image owns the binary, an upgrade replaces it completely, and nothing writes it at runtime. Decision 4 follows from it. A Kubernetes upgrade is a new image, bootc upgrade, a reboot, and an atomic rollback if the upgrade goes wrong. The result is one upgrade mechanism, one version axis, and free rollback.

That is a good trade for the OS. It is a bad trade for Kubernetes, because the two change for different reasons and at different rates.

An operator on Corium 0.3.6 runs Kubernetes 1.36.4, because 0.3.6 contains that version. To get 1.37, the operator waits for a Corium release that carries it. To get a Fedora security fix, the operator takes the Kubernetes minor that this Corium release carries, whether or not the workloads are ready for it. Today, neither direction can be separated, and both are ordinary requests. The coupling also affects the project. Every Kubernetes minor that Corium wants to offer is a Corium release. Every Corium release is a statement about a Kubernetes version that Corium did not intend to make.

Decision 5 makes this worse. k0s Autopilot is the upstream answer to in-place Kubernetes upgrades. It is disabled, correctly, because it mutates the k0s binary in place, and decision 3 says that this binary is image content. So Corium disables the mechanism and does not replace it. An operator who wants a different Kubernetes version on a running node has two options left. The first is cctl reset, which removes the node from its cluster. The second is a full OS rebase to an image that the operator possibly does not want for any other reason.

What Talos does, and why it does not transpose#

Talos separates talosctl upgrade from talosctl upgrade-k8s. The OS image contains no Kubernetes version at all. The control plane runs as static pods from registry images. The kubelet is a container image named per node. An upgrade rewrites image tags in the machine configuration. A support matrix sets which Kubernetes versions each Talos release accepts.

The shape is right, but the mechanism does not transfer. k0s is a single self-extracting executable: a binary with a ZIP payload appended. The payload carries kubelet, containerd, runc, the API server, scheduler, controller-manager, etcd and the CNI plugins. On start, k0s stages these contents into /var/lib/k0s/bin (pkg/assets/stage.go). There is no set of image tags to rewrite. To decouple the Kubernetes version from the OS version is to decouple that file from the OS image, and nothing else.

What the platform offers#

• systemd system extensions merge /usr and /opt with an overlayfs. The man page explicitly names immutable system images as the use case. The merged /usr stays read-only. systemd searches for extensions in /etc/extensions/, /run/extensions/ and /var/lib/extensions/. systemd documents the last one as the primary install location. On an OSTree system, it is also the only writable one. systemd-sysext.service is ordered before basic.target. So an ordinary unit such as k0scontroller.service sees the merged tree, and Corium does no extra work for it. ( systemd-sysext(8))
• Each extension declares what it fits. /usr/lib/extension-release.d/extension-release.<NAME> must match the ID of the host. If the extension sets SYSEXT_LEVEL, that value must match. If not, VERSION_ID must match. This is the sharpest edge in the whole design, and the decision below treats it as such.
• The pattern is established, but not for k0s. The Fedora image-mode sysext project publishes kubernetes-1.33 through kubernetes-1.36. The Flatcar sysext-bakery publishes Kubernetes, containerd and Docker extensions with systemd-sysupdate metadata. Flatcar states plainly that it supports updates within a minor and never across one. Neither catalog contains k0s. So Corium cannot consume this pattern. It must produce it. ( fedora-sysexts, sysext-bakery)
• k0s supports the swap. k0s stop, replace the binary, k0s start is the documented upgrade path without Autopilot. The unit that k0s install writes puts the absolute path of the binary into ExecStart. Usually, that makes a swap a reinstall. But here the path does not change, because an extension overlays /usr/bin/k0s and does not install a replacement at another path. ( k0s upgrade)
• k0s’s version skew rules are stricter than upstream Kubernetes’. Controllers must be within one minor of each other. A worker can be one minor older than the controllers, and never newer. Upgrades go one minor at a time, controllers first. At bootstrap, a worker must match its controllers exactly. ( version skew policy)

Decision#

/usr/bin/k0s stops being image content and becomes a slot. The path does not change, and neither does any caller of it: internal/k0s.Binary keeps its value. What changes is who fills the slot.

1. The image still ships a k0s, and an extension overlays it#

The OS image continues to install a pinned k0s at /usr/bin/k0s, exactly as it does today. That binary is the floor: the version that a node runs when nobody told it a version, when it cannot reach a mirror, or when its extension stopped matching.

This is the most important part of the decision, and it is plain on purpose. To ship no k0s at all is cleaner on paper. But then first-boot network access to a mirror becomes necessary for a machine to become a node. The sharpest failure mode (an extension that does not load) then gives a dead node, not a degraded one. It also makes this change breaking for every node already in service. The floor costs the image size of one binary. In return, a node always boots into some Kubernetes.

2. Extensions are signed OCI artefacts, pulled and verified like the OS#

Corium publishes each k0s system extension as a signed OCI artifact, referenced by digest. The existing container signing policy of the node is the gate. It is the same policy that internal/upgrade/policy.go already applies before it lets bootc move a node to another OS image: an artifact is acceptable only if /etc/containers/policy.json demands a signature for it.

This is reuse, not symmetry for its own sake. Corium already refuses to move a node onto an unsigned OS image. A file that becomes the kubelet and the API server on that node deserves the same refusal. A bare URL and a checksum give integrity without provenance. They protect against corruption in transit, but not against the source of the artifact. build/k0s.lock already draws that distinction in its own header.

Amended 2026-09-26: skopeo ships in the image. When this record was written, the promise above had no mechanism behind it. A Corium node carries no container engine, and the Containerfile says so on purpose. So nothing on the node was able to pull an artifact. Nothing was able to apply policy.json either, because policy.json is not a file that the node interprets. It is a contract that the containers/image library implements. Without code that reads it, a promise of verification “by the same policy.json” made this section only an aspiration.

skopeo is the command line of that library, and this amendment adds it to the image for that purpose. Be precise about what it is not. It copies images between registries and local layouts, and it runs nothing. The Containerfile objects to a container engine because a second engine fights k0s for cgroups and CNI state. That objection does not apply to skopeo. It has no daemon, no runtime, and nothing that starts at boot.

The alternative was to write a registry client and sigstore verification in Go. Decision 13 is the argument against it: signature verification is exactly the class of format whose hand-rolled implementation “is wrong in ways nobody notices until a node trusts a key it should not”. This is the shape of exception that the rule allows, and it is a package in an image, not an entry in go.mod.

3. A new corium.kubernetes: block#

corium:
  kubernetes:
    version: v1.36.4+k0s.0
    mirror: registry.internal.example/corium/k0s

Both fields are optional, as the twenty-line constraint of decision 1 requires. If version is not set, the node runs the floor. mirror redirects the pull, and this is how an air-gapped site is served. Point mirror at an internal registry, instead of asking Corium to invent an offline mode.

Validation is offline, per AGENTS §7. It checks the syntax of the version, its presence in the support window that the image was built with, and the skew rules, all without a network call. The pull answers whether the artifact exists.

4. The support window is three minors#

An image supports the Kubernetes minor that it ships as its floor and the two adjacent minors. That is three in total, published as a matrix and stated in the docs. The number three comes from the skew rules of k0s, not from taste. The rules are: one minor at a time, controllers first, and workers no newer than controllers. So a cluster in the middle of a rollout legitimately spans two minors. A third minor gives an operator somewhere to go, without first taking an OS upgrade that the operator did not ask for.

The window is a property of the OS image. To move outside it, an operator still needs a bootc upgrade. The two axes are independent, but they are not unbounded.

5. Matching is by SYSEXT_LEVEL, which Corium owns#

An extension that pins VERSION_ID binds itself to a Fedora major. bootc upgrade does not touch /var, by design. So an extension in /var/lib/extensions survives an OS rebase and then silently stops matching. A node that rebases from Fedora 44 to 45 comes back on the floor version, or on nothing, with no warning. The Fedora sysext project lists this as a known issue.

So extensions declare a SYSEXT_LEVEL that Corium defines and versions on purpose. The cost is real, and this record accepts it. Corium takes ownership of an ABI contract between its OS images and its k0s extensions. A bump of that level is a reviewable event, in the same way that a change to build/k0s.lock is today.

6. Day two: kubernetes.version joins ADR 8’s safe subset, behind a skew gate#

ADR 8 established that a bootstrapped node re-applies the fields whose reconciler Corium already delegates to, and refuses everything that defines what the node is. kubernetes.version joins addons and k0s.patch in the safe subset. It does not define what the node is. The node stays the same machine, in the same cluster, under the same name and address. It only runs a different build of the software that serves it.

It carries a constraint that the other two fields do not carry, and PlanReconcile enforces it. If a change violates the skew rules of k0s, PlanReconcile refuses it and names the rule. It does not pass the change to k0s to discover at restart. Skipping a minor is the error that operators actually make, and the moment to report it is before anything moves.

A version change is a bounded sequence that the node performs on itself: drain, k0s stop, swap the active extension, clear /var/lib/k0s/bin, k0s start, uncordon. The purge is not tidiness. k0s skips re-staging its embedded payload when the mtime of the already-staged file equals the mtime of the k0s executable and its size matches. An EROFS extension built reproducibly has a normalized mtime. So two versions can collide, and a node can run a stale kubelet against a new control plane. The purge removes this possibility, so nobody has to reason about when it can happen.

7. Ordering a rollout stays with the operator#

cctl gets the verb. The daemon does not get a rollout. A node knows only about itself, and “controllers first, one at a time, then workers” is a statement about the other nodes. This is exactly the division that decision 11 and ADR 4 draw, and the one that internal/cctl/rollout.go already implements for OS upgrades.

8. Autopilot stays disabled#

This record does not weaken decision 5. Decision 5 is what makes this record coherent. Autopilot was rejected because it mutates the binary in place, without the knowledge of the operator. Corium now offers the capability that Autopilot was wanted for, a Kubernetes version change without a reset, as an explicit, operator-triggered, signed, drain-aware act. The objection was never to in-place upgrades. It was to a controller that performs them without being asked.

Consequences#

Decision 4 is split, and must be read as two. OS upgrades stay image-based with atomic rollback, and they do not change. Kubernetes upgrades become their own axis with their own verb. “One upgrade mechanism, one version axis, free rollback” was worth stating while it was true. This record is where it stops being true. Rollback of a Kubernetes version is no longer free. To revert to the previous extension is a second forward operation, and k0s does not support a downgrade of a minor at all. The docs must say this plainly, and not leave operators to discover it.

Decision 3 is amended in letter, kept in spirit. The binary is no longer image content. But it is still read-only content from a signed, digest-pinned artifact that nothing on the node writes to. Corium does not trade “Immutability is the product” for a mutable /usr/bin/k0s. It extends that rule to cover a second immutable source.

Corium becomes a publisher of k0s artifacts. Nobody publishes k0s system extensions, so Corium builds, signs and hosts them. This is a new supply-chain surface and new release work: three minors, two architectures, signed and published on their own schedule. It is the largest single cost in this record. It also means that build/k0s.lock stops being a file about one version and becomes a file about a window.

Two version axes are two things to explain. A support matrix must exist, be published, and be correct. cctl must answer “what Kubernetes am I running, and what can I move to”, so that an operator does not have to cross-reference a table. The twenty-line constraint holds, because both new fields are optional. But the number of concepts went up by one. That is a real cost to a project whose stated goal is that a node is describable in twenty lines.

A new failure mode needs to be visible. An extension that stops matching does not fail loudly. It does not merge, and the node comes back on the floor version. Greenboot must check that the running k0s version is the declared one, and fail the boot if it is not. Otherwise, the quietest error of the design becomes its most common support question.

First boot can now depend on the network. A node that declares a non-floor version must reach its mirror. The floor keeps this from being fatal. But a node that boots into 1.36 when its configuration said 1.37 did not do what it was asked. It must say so, and not appear healthy.

Alternatives considered#

Download the binary into /var at bootstrap. This is the obvious implementation, and the reason to write this record instead. It puts a mutable, unsigned file on the most critical path of the node. It gives up the read-only guarantee for the one binary that needs it most. It also reimplements, badly, the verification that policy.json and the sysext machinery already provide. It is cheap to build and expensive to own.

Build image variants: corium:0.4.0-k8s1.33, -k8s1.34, … This keeps decisions 3 and 4 completely, which is its appeal. It is rejected because it does not actually decouple the axes. It multiplies them. The OS × Kubernetes matrix applies to every build, every signature, every publish and every test. An operator who changes only the Kubernetes minor still needs a full OS rebase and a reboot to do it. This option moves the coupling into the tag and calls it a separation.

Ship several k0s binaries inside /usr and select one at boot. This keeps every existing guarantee and needs no network at all, which is a real advantage. It is rejected for two reasons. First, the image still bounds the window. To reach a fourth minor is still an OS upgrade, so the coupling is looser but not broken. Second, three self-extracting k0s binaries are a large dead weight in every image, on every node, and most of that weight never runs. It is the floor idea taken to three floors, and the third floor does not pay for itself.

bootc logically bound images. This is the mechanism built to pin content to an OS image, so it deserves serious consideration. It is rejected on a fact. Logically bound images (LBIs) deliver podman containers, pulled at install and upgrade time, with no mechanism to put a file in /usr/bin. To reach the binary, the node must run a container that copies its contents onto the host. That is a sysext with extra steps and no read-only guarantee. The ecosystem is also not yet stable. Anaconda does not support LBIs, and image-builder has open issues where bound images fail against the default store of podman.

Enable k0s Autopilot. This is the upstream answer, and it is available today. It closes the capability gap, and Corium does not have to publish anything. It is rejected for the reason that decision 5 gives and this record keeps. It mutates /usr/bin/k0s on a running node from a controller in the cluster. That violates the read-only contract, and it is a fleet-management posture that decision 11 declines. A cluster resource that rewrites the operating system is the wrong direction of control.

Status quo. The status quo stayed in place too long, but it was defensible. One axis is truly simpler. A project below 1.0 with a small support surface can credibly say “take the image that has the Kubernetes you want”. It stops being defensible when Corium has operators who need a Fedora security fix without a Kubernetes minor bump, or a Kubernetes minor bump without a Fedora rebase. Both needs are ordinary, and the image axis can express neither.