ADR 6 — Host WireGuard overlay

Status: accepted

Context#

Issue #7 asks for a way to declare a WireGuard interface on the node itself, from the corium: block. With it, nodes across sites or providers can form one cluster over an encrypted overlay (a private network that runs on top of the public one). The issue carries the same caveat that issue #5 carried for RAID: add the feature only if it is better than a plain cloud-init WireGuard setup. Decision 9 in AGENTS.md is the standing rule behind that caveat. Every abstraction has an escape hatch, and write_files plus runcmd are always available. So the field must justify itself. The escape hatch does not have to justify its absence.

Three facts must be established before the decision means anything. Each of them changes what “the plain cloud-init alternative” actually is on this operating system.

The problem is a host problem, and the CNI does not solve it. k0s can already encrypt pod-to-pod traffic. kube-router and Calico both offer a WireGuard or IPsec mode. But that encryption operates between pods, and it assumes that the hosts can already reach one another. Across sites or providers, the hosts cannot. The only address that every node shares is its public address. Without an overlay, the traffic of the control plane itself crosses the open internet: kubelet to API server, etcd peer replication, the konnectivity tunnel, and the join API. A host overlay gives every node a stable address in one private range, and k0s runs over it. That is a different problem from the one that the CNI solves, and this feature exists for it. Without that problem, the feature has no case. The problem is real.

The tooling is an image concern, and half of it is already met. On a mutable distribution, “a plain cloud-init WireGuard setup” starts with packages: [wireguard-tools]. On Corium, that step does nothing. /usr is read-only and the package set is fixed at build time. 05-corium.cfg disables package_upgrade/package_update on purpose, so that a first-boot install fails early with a clear error, not late with an unclear one. So the tooling cannot be a boot concern at all. It is an image concern. This is a checked fact about the base image, not an assumption. The kernel module is already there. The Fedora 44 base carries wireguard.ko.xz under /usr/lib/modules/<kver>/kernel/drivers/net/wireguard/. The module is present but not loaded, and that is the disabled state. A modules-load.d entry only makes the load early and deterministic. The userspace half is not there. rpm -q wireguard-tools on quay.io/fedora/fedora-bootc:44 reports that the package is not installed. So wg, wg-quick and the wg-quick@.service template are all absent. The decision is therefore to make them present: add wireguard-tools to the Corium image, present but with nothing enabled. This is the same posture that ADR 5 takes for sshd (present, but with an empty guest list). A wg-quick@.service template that nobody instantiates is not a running service. So the cost is one small, dormant package, not a listening surface. The escape-hatch recipe needs that same one line anyway.

This fact comes first because it removes the tooling from the comparison between a field and a recipe. Both use the same image with the same dormant tools. Three other things separate them: where the configuration can come from, which address the node registers, and where the private key is stored. This ADR argues from those three things.

NetworkManager is the only renderer. 05-corium.cfg records that the image runs NetworkManager and that nothing else manages the network. systemd-networkd is not the active renderer. So a .netdev/.network approach means a second network daemon next to NetworkManager. That gives two authorities over the same interfaces, which is the type of problem that decision 7 rejected Ignition to avoid. Two paths to create the interface remain: wg-quick that reads /etc/wireguard (needs wireguard-tools), or a NetworkManager keyfile connection. The keyfile is native and needs no extra package, but it couples the design to NM and still writes the key into /etc. The decision below picks one path and records the other as an alternative.

Decision#

Add a wireguard: block to the corium: schema. The block describes this node’s participation in a host overlay: one or more interfaces, each with its address, listen port, private key (inline or resolved at boot), and peers. corium-agent provisions it at first boot, before k0s, in the same way as raid[].

The first reason to prefer the field is where it can be read from. A recipe is made of write_files and runcmd. These are cloud-init modules, and they run only where cloud-init runs. But Corium does not depend on cloud-init for its configuration, by design. internal/source resolves the corium: block from a chain: /etc/corium/config.yaml first, then cloud-init, then the kernel command line, then an image default. The chain exists because “bare metal without a seed device, PXE, and pre-baked appliances all need a configuration and have no datasource.” For this feature, that is not an edge case. It is the main case. An overlay exists to join nodes across sites and providers. That is where on-premise and bare-metal nodes without a cloud-init datasource are most common. On those nodes, the recipe never runs. A corium: field is read from /etc/corium/config.yaml or the kernel command line like any other field. The escape hatch is weakest in exactly the environment that the feature targets.

Beyond that reach, the field owns three things that a recipe cannot own well. Together with the reach, they are the whole justification:

1. The interface is up before k0s starts, and stays a boot-time artifact. corium-agent runs only on first boot. The overlay must survive every reboot after that, and it must be up before the kubelet registers. So the agent creates a persistent unit: wg-quick@<name>.service over an /etc/wireguard/<name>.conf file that the agent writes. The agent enables the unit. It brings the interface up synchronously during bootstrap, before it renders k0s or detects the node address. It then writes a drop-in on the k0s unit that orders k0s After=/Requires= the interface. This reuses the same mechanism that internal/bootstrap/raid.go uses to make k0s wait for a mount (requireMountsForK0s). If the overlay of a node is down, the node does not half-join a cluster. k0s refuses to start and gives the reason.

2. The overlay address becomes the address the node registers. This is the nodeip.go failure class, and it is the reason why a recipe is not enough. Today, --node-ip is passed only when HA is enabled. For a plain cross-site cluster, the kubelet picks the first address that the interface scan finds. That address is on the physical NIC, not on the overlay. The node then registers an address that its peers cannot reach, and logs, exec, port-forward and metrics all fail. This is the same delayed, confusing failure that detectNodeIP was written to prevent for the VRRP virtual IP. A cloud-init runcmd cannot fix it, because the choice of address happens inside corium-agent, after cloud-init finished. The operator can mark one interface as the cluster address of the node. corium-agent registers that address with k0s. On a controller, the address is also added to the SANs of the API server. When HA also runs over the overlay, detectNodeIP selects from that interface, not from the default route.

3. The private key is handled as a secret. A write_files recipe puts the WireGuard private key in cleartext into the instance metadata. join.tokenFrom, ha.authPassFrom and api.operatorCAFrom exist to prevent exactly that. The field reuses SecretSource. privateKeyFrom resolves the key at boot from an HTTPS URL or a local file, with the existing waitFor retry semantics, and internal/secret never logs it. corium-agent writes the rendered /etc/wireguard/<name>.conf with mode 0600, owned by root, and redacts the key in all of its reports. The public key of a peer is not a secret and stays inline. A peer preshared key is a secret, and gets the same ...From treatment.

Schema sketch#

These are not the final types. This is a sketch of the surface, to discuss its shape, not to implement it. It is a list, like raid[], because a node can in principle carry more than one interface. One interface is the expected case.

corium:
  role: controller+worker
  cluster:
    # The overlay address, so kubeconfigs and joining nodes use the tunnel.
    endpoint: 10.10.0.1
  wireguard:
    - name: wg0                 # becomes the interface name; /dev-style unique per node
      address: 10.10.0.1/24     # this node's address on the overlay (CIDR)
      listenPort: 51820         # optional; omit for a client-only node
      nodeAddress: true         # this interface carries the address k0s registers
      mtu: 1420                 # optional
      privateKeyFrom:           # SecretSource: exactly one of url/file, never inline in prod
        file: /run/corium/wg0.key
        waitFor: 10m
      peers:
        - publicKey: "base64=="            # not a secret; inline
          endpoint: controller.example:51820
          allowedIPs: [10.10.0.0/24]
          persistentKeepalive: 25          # optional; needed behind NAT
          presharedKeyFrom:                # SecretSource; optional, defence in depth
            file: /run/corium/wg0.psk

An inline privateKey: field exists next to privateKeyFrom for labs. It mirrors join.token/join.tokenFrom. The documentation calls it a liability for the same reason: anything that can read the instance metadata can read the key.

Validation rules#

Validation is offline and reports all problems at once with errors.Join. This is the same contract as the rest of Validate(): no network, no filesystem, and every problem reported in one boot.

• wireguard[].name: required, unique across the block, and a valid Linux interface name (^[a-z0-9][-a-z0-9]*$, ≤ 15 characters, the IFNAMSIZ limit).
• wireguard[].address: required, a valid CIDR (netip.ParsePrefix). This is a host address with a prefix length, not a bare IP.
• wireguard[].listenPort: if set, 1–65535. It must be set on any node that a peer names in its endpoint, because nobody can dial a node with no listen port.
• wireguard[].mtu: if set, in the range 576–65535. If it is not set, the tool picks the default.
• privateKey / privateKeyFrom: set exactly one. Reject both, and reject neither. SecretSource.validate already covers the ...From half (https-only URL, absolute file path, bounded waitFor). Validation checks an inline key for base64 shape and 32-byte length. So a truncated paste fails at validation, not at wg runtime.
• nodeAddress: at most one interface across the block can set it to true. If no interface sets it and the node joins over the overlay, that is a likely mistake. Validation warns in the “settings that do nothing” style that it already uses for a disabled ha: block. The reason is that a node whose registered address is not its overlay address looks healthy and is unreachable.
• peers[].publicKey: required, base64, 32 bytes. Validation rejects duplicate public keys within one interface. A duplicate is a copy-paste error that silently drops a peer.
• peers[].endpoint: if set, host:port. The host can be a name or an address, but an IP is recommended. A name must resolve before the network is up, and that network can depend on the overlay itself.
• peers[].allowedIPs: required, each a valid CIDR. Validation rejects overlapping allowedIPs across peers on one interface. WireGuard routes by longest match. Two peers that claim the same range are a routing ambiguity, not a configuration that an operator can intend.
• peers[].persistentKeepalive: if set, 1–65535 (seconds).
• If wireguard: is empty, the block does nothing and is silent. A wireguard: block on a node whose role never joins anything is not an error by itself, because an overlay can exist for reasons other than the cluster.

Consequences#

A permanent, non-trivial schema surface. The package doc in types.go says it plainly: a key added here is permanent, ships to every node, and must be supported forever, so add them reluctantly. An interface with peers is more surface than any single key added so far, and this ADR spends that surface on purpose. What the block is not keeps the scope small. It describes the view of one node on an overlay, not a mesh. It creates no keys, distributes none, and rotates none. It knows nothing of the other nodes beyond the peers that it is given. Key material and topology management across a fleet are a control-plane concern. Corium provisions nodes and does not manage them (a non-goal), and that line keeps this block from growing into a control plane.

A third deliberate /etc exception. wg-quick reads /etc/wireguard, as sshd reads /etc/ssh/sshd_config.d and cloud-init reads /etc/cloud. The justification is the same as in ADR 5: it is the only place that the tool reads from, so the configuration must live there or nothing reads it. This ADR names it as an exception and does not slip it in silently. The file is written with mode 0600 because it carries the private key. The sshd drop-in can be world-readable, but this file cannot.

wireguard-tools added to the image, present on every node, used by few. The package is not in the base image, and this was verified. So to accept this decision is to add the package to the Corium Containerfile. It then ships whether or not a node ever declares an interface. It is a small, dormant package on machines that never use it, which is the cost of any package in an OS image. This is on purpose the same bargain that ADR 5 made for sshd: present and inert until something turns it on. The other choice is a package that is absent and cannot be turned on, on an immutable OS. The module is already in the base kernel. For each release, re-check the package addition and the wg-quick@.service template that it brings with rpm -q, exactly as ADR 5 pinned the inherited sshd.

corium-agent learns to configure a host interface. This is the first host network interface that Corium declares. Until now, host networking belonged to cloud-init alone, and the corium: block described only cluster addressing (pod/service CIDRs, the VRRP VIP). That boundary moves a little here. The ADR names the move so that the next person does not have to discover it again. detectNodeIP and the HA path both learn about the overlay interface. A WireGuard overlay that carries a VRRP VIP works only over unicastPeers, because multicast does not cross a point-to-point tunnel. detectNodeIP must select from the overlay interface, not from the default route. These are real changes to the most delicate code in the tree, and their tests are the price of the feature.

One window this record will not pretend is closed. A peer endpoint given as a hostname must resolve before the overlay is up. If DNS itself is reachable only over the overlay, that is a cycle that the field cannot break. Validation warns, and the documentation says to use addresses. The field does not make a hostname endpoint safe. It only makes the failure clear.

Alternatives considered#

Documentation only: a cloud-init recipe, no field. This is the honest baseline. It is rejected for three reasons that a recipe cannot fix. First, it runs only where cloud-init runs, because write_files and runcmd are cloud-init modules. Some on-premise or bare-metal nodes have no datasource. They get their configuration from /etc/corium/config.yaml or the kernel command line, the paths that internal/source exists to serve. On those nodes, the recipe never runs at all, and those nodes are the exact population that an overlay across sites and providers is for. Second, the recipe cannot make the overlay address the address that the kubelet registers. corium-agent makes that choice after cloud-init ran, and no kubelet-args escape hatch lets a runcmd reach it. Third, the recipe cannot keep the private key out of cleartext instance metadata. Everything else about a recipe works. A write_files configuration and an enabled wg-quick@.service do bring an interface up before k0s. But the three gaps turn a working demo into one of three failures: a node that is unconfigured off-cloud, a node that is quietly unreachable from across the overlay, or a cluster whose keys leak with its metadata. ADR 3 used the same reasoning to reject documentation-only for spare-disk RAID: the ordering problem is the one that silently breaks a node. The recipe will still be documented as the escape hatch (decision 9). It is still the right tool for an overlay that is not the cluster transport, on a node that has cloud-init anyway.

A bare node-IP override field plus the recipe. This is tempting as the smaller move: ship wireguard-tools, add a general node.address override so that the kubelet registers the address it is given, and document the interface as a recipe. It is rejected because it splits one invariant, “the node registers its overlay address”, across two places. The operator must keep the two in sync by hand, and the agent cannot check that the interface actually carries the address that the field names. It also leaves the private key in the write_files of the recipe, unsolved. ADR 3 was able to split its problem cleanly because there was a real seam: spare disks are declarable at first boot, and a RAID root is impossible at first boot. Here there is no such seam. Interface bring-up, node addressing and the secret all apply at first boot and after, all sit close to the agent, and all involve the key. So a split cuts through the middle of one problem, not between two problems.

A NetworkManager keyfile connection instead of wg-quick. NetworkManager manages WireGuard natively, and it is the one renderer that the image already runs. So it can create the interface without wg-quick at all. It is rejected as the primary path for two reasons. First, it couples the feature to NM as the renderer forever, but wg-quick is renderer-agnostic and survives a future change of mind. Second, wireguard-tools ships present but inert in all cases, because wg is wanted for diagnostics on a node whose overlay misbehaves. So the package that NM saves is one that Corium keeps anyway, and avoiding wg-quick gains little. The schema does not depend on which of the two creates the interface. So this stays an implementation choice that the ADR does not need to freeze.

Managing the overlay’s keys and mesh from the corium: block. No. To create and distribute keys, and to know the whole topology, is fleet management. Corium provisions nodes and does not manage them. The block describes the peers of one node and nothing more.