ADR 6 — Host WireGuard overlay

Status: accepted

Context#

Issue #7 asks for a way to declare a WireGuard interface on the node itself, from the corium: block, so that nodes across sites or providers can form one cluster over an encrypted overlay. It carries the same caveat issue #5 did for RAID: only add it if it earns its place over a plain cloud-init WireGuard setup. Decision 9 in AGENTS.md is the standing rule behind that caveat — every abstraction has an escape hatch, and write_files plus runcmd are always there — so the burden is on the field to justify itself, not on the escape hatch to justify its absence.

Three facts have to be established before the decision means anything, because each of them changes what “the plain cloud-init alternative” actually is on this operating system.

The problem is a host problem, and the CNI does not solve it. k0s can already encrypt pod-to-pod traffic — kube-router and Calico both offer a WireGuard or IPsec mode — but that operates between pods and takes for granted that the hosts can already reach one another. Across sites or providers they cannot: the only address every node shares is its public one, and the control plane’s own traffic — kubelet to API server, etcd peer replication, the konnectivity tunnel, the join API — would otherwise cross the open internet. A host overlay gives every node a stable address in one private range and lets k0s run over it. That is a different problem from the one the CNI solves, and it is the problem this feature exists for. If there were no such problem the feature would have no case; there is one.

The tooling is an image concern, and half of it is already met. On a mutable distribution “a plain cloud-init WireGuard setup” starts with packages: [wireguard-tools]. On Corium that step is a no-op: /usr is read-only, the package set is fixed at build time, and 05-corium.cfg disables package_upgrade/package_update precisely so a first-boot install fails early and legibly instead of late and mysteriously. So the tooling cannot be a boot concern at all — it is an image concern, and this was checked against the base image rather than assumed. The kernel module is already there: the Fedora 44 base carries wireguard.ko.xz under /usr/lib/modules/<kver>/kernel/drivers/net/wireguard/, present-but-unloaded, which is the disabled state; a modules-load.d entry would only make the load early and deterministic. The userspace half is not there — rpm -q wireguard-tools on quay.io/fedora/fedora-bootc:44 reports it not installed, so wg, wg-quick and the wg-quick@.service template are all absent. Making it present is therefore the decision: add wireguard-tools to Corium’s own image, present but with nothing enabled, exactly the posture ADR 5 takes for sshd (present, but with an empty guest list). A wg-quick@.service template that is never instantiated is not a running service, so the cost is one small, dormant package, not a listening surface — and it is the same one line the escape-hatch recipe would need anyway.

The point of stating this first is to take the tooling off the table as the thing that distinguishes a field from a recipe. Both stand on the same image with the same dormant tools. What separates them is everything below: where the configuration can come from, which address the node registers, and where the private key lives. Those are the comparison this ADR argues.

NetworkManager is the only renderer. 05-corium.cfg records that the image runs NetworkManager and nothing else manages the network. systemd-networkd is not the active renderer, so a .netdev/.network approach would mean running a second network daemon alongside NetworkManager — two authorities over the same interfaces, which is the shape of problem decision 7 rejected Ignition to avoid. That leaves two materialisation paths worth considering: wg-quick reading /etc/wireguard (needs wireguard-tools), or a NetworkManager keyfile connection (native, no extra package, but couples the design to NM and still writes the key into /etc). The decision below picks one and records the other as an alternative.

Decision#

Add a wireguard: block to the corium: schema, describing this node’s participation in a host overlay: one or more interfaces, each with its address, listen port, private key (inline or resolved at boot), and peers. It is provisioned by corium-agent at first boot, before k0s, on the same footing as raid[].

The first reason to prefer the field is where it can be read from. A recipe is made of write_files and runcmd, which are cloud-init modules and run only where cloud-init runs. But Corium deliberately does not depend on cloud-init for its configuration: internal/source resolves the corium: block from a chain — /etc/corium/config.yaml first, then cloud-init, then the kernel command line, then an image default — precisely because “bare metal without a seed device, PXE, and pre-baked appliances all need a configuration and have no datasource.” That is not a corner case for this feature; it is its centre. An overlay exists to join nodes across sites and providers, which is where on-premise and bare-metal nodes without a cloud-init datasource are most common — and on those nodes the recipe simply never executes, while a corium: field is read from /etc/corium/config.yaml or the kernel command line like any other. The escape hatch is weakest in exactly the environment the feature targets.

Beyond that reach, the field owns three things a recipe cannot own well, and together with it they are the whole justification:

1. The interface is up before k0s starts, and stays a boot-time artefact. corium-agent runs only on first boot; the overlay has to survive every reboot after that and be up before the kubelet registers. So the agent materialises a persistent unit — wg-quick@<name>.service over an agent-written /etc/wireguard/<name>.conf — enables it, and brings the interface up synchronously during bootstrap before it renders k0s or detects the node address. It then writes a drop-in onto the k0s unit ordering it After=/Requires= the interface, reusing exactly the mechanism internal/bootstrap/raid.go already uses to make k0s wait for a mount (requireMountsForK0s). A node whose overlay is down does not half-join a cluster; k0s refuses to start and says why.

2. The overlay address becomes the address the node registers. This is the nodeip.go failure class, and it is the reason a recipe is not enough. Today --node-ip is passed only when HA is enabled; for a plain cross-site cluster the kubelet picks whatever address the interface scan finds first, which is the physical NIC, not the overlay. The node then registers an address its peers cannot reach, and logs, exec, port-forward and metrics all go nowhere — the same delayed, confusing failure that detectNodeIP was written to prevent for the VRRP virtual IP, and one a cloud-init runcmd cannot fix because the address is chosen inside corium-agent, after cloud-init has finished. An interface the operator marks as the node’s cluster address is the address corium-agent registers with k0s, is added to the API server’s SANs on a controller, and is the address detectNodeIP selects from (rather than the default route) when HA also runs over the overlay.

3. The private key is handled as a secret. A write_files recipe puts the WireGuard private key in cleartext into the instance metadata — the precise thing join.tokenFrom, ha.authPassFrom and api.operatorCAFrom exist to avoid. The field reuses SecretSource: privateKeyFrom resolves the key at boot from an HTTPS URL or a local file, with the established waitFor retry semantics, and internal/secret never logs it. corium-agent writes the rendered /etc/wireguard/<name>.conf 0600, root-owned, and redacts the key everywhere it reports. A peer’s public key is not a secret and sits inline; a peer preshared key is, and takes the same ...From treatment.

Schema sketch#

Not final types — a sketch of the surface, to argue about shape rather than to implement. A list, like raid[], because a node may in principle carry more than one interface, though one is the expected case.

corium:
  role: controller+worker
  cluster:
    # The overlay address, so kubeconfigs and joining nodes use the tunnel.
    endpoint: 10.10.0.1
  wireguard:
    - name: wg0                 # becomes the interface name; /dev-style unique per node
      address: 10.10.0.1/24     # this node's address on the overlay (CIDR)
      listenPort: 51820         # optional; omit for a client-only node
      nodeAddress: true         # this interface carries the address k0s registers
      mtu: 1420                 # optional
      privateKeyFrom:           # SecretSource: exactly one of url/file, never inline in prod
        file: /run/corium/wg0.key
        waitFor: 10m
      peers:
        - publicKey: "base64=="            # not a secret; inline
          endpoint: controller.example:51820
          allowedIPs: [10.10.0.0/24]
          persistentKeepalive: 25          # optional; needed behind NAT
          presharedKeyFrom:                # SecretSource; optional, defence in depth
            file: /run/corium/wg0.psk

An inline privateKey: field exists alongside privateKeyFrom for labs, mirror of join.token/join.tokenFrom, and is documented as a liability for the same reason: anything that can read the instance metadata can read it.

Validation rules#

Offline, all-at-once, errors.Join — the same contract as the rest of Validate(): no network, no filesystem, every problem reported in one boot.

• wireguard[].name: required, unique across the block, and a valid Linux interface name (^[a-z0-9][-a-z0-9]*$, ≤ 15 characters — IFNAMSIZ).
• wireguard[].address: required, valid CIDR (netip.ParsePrefix). This is a host address with a prefix length, not a bare IP.
• wireguard[].listenPort: if set, 1–65535. Required to be set on any node a peer names in its endpoint (a node with no listen port cannot be dialled).
• wireguard[].mtu: if set, a sane range (576–65535); default left to the tool.
• privateKey / privateKeyFrom: set exactly one; reject both, and reject neither. SecretSource.validate already covers the ...From half (https-only URL, absolute file path, bounded waitFor). An inline key is checked for base64 shape and 32-byte length, so a truncated paste fails at validation rather than at wg runtime.
• nodeAddress: at most one interface across the block may set it true. If none does and the node joins over the overlay, that is a likely mistake — warn in the “settings that do nothing” style already used for a disabled ha: block, because a node whose overlay is not its registered address will look healthy and be unreachable.
• peers[].publicKey: required, base64, 32 bytes. Duplicate public keys within one interface are rejected (a copy-paste error that silently drops a peer).
• peers[].endpoint: if set, host:port; the host may be a name or an address, but an IP is recommended, since a name has to resolve before the network the overlay itself may gate is up.
• peers[].allowedIPs: required, each a valid CIDR. Overlapping allowedIPs across peers on one interface are rejected — WireGuard routes by longest match and two peers claiming the same range is a routing ambiguity, not a configuration two people can both have meant.
• peers[].persistentKeepalive: if set, 1–65535 (seconds).
• If wireguard: is empty the block does nothing and is silent; a wireguard: block on a node whose role never joins anything is not itself an error, since an overlay can exist for reasons other than the cluster.

Consequences#

A permanent, non-trivial schema surface. The package doc in types.go says it plainly: a key added here is permanent, ships to every node, and must be supported forever — add them reluctantly. An interface with peers is more surface than any single key added so far, and this ADR spends it on purpose. The scope is held down by what the block deliberately is not: it describes one node’s view of an overlay, not a mesh. It mints no keys, distributes none, rotates none, and knows nothing of the other nodes beyond the peers it is told about. Key material and topology management across a fleet is a control-plane concern, and Corium provisioning nodes rather than managing them (a non-goal) is the line that keeps this from growing into one.

A third deliberate /etc exception. wg-quick reads /etc/wireguard, as sshd reads /etc/ssh/sshd_config.d and cloud-init reads /etc/cloud. The justification is the same as ADR 5’s: it is the only place the tool reads from, so the config has to live there or not be read. It is named as an exception rather than slipped in, and the file is written 0600 because it carries the private key — where the sshd drop-in could be world-readable, this cannot.

wireguard-tools added to the image, present on every node, used by few. The package is not in the base — this was verified — so accepting this decision means adding it to Corium’s Containerfile, where it then ships whether or not a node ever declares an interface: a small, dormant package on machines that will never use it, which is the cost of any package in an OS image. It is deliberately the same bargain ADR 5 struck for sshd: present and inert until something turns it on, rather than absent and un-turn-on-able on an immutable OS. The module is already present in the base kernel; the package addition and the wg-quick@.service template it brings should be re-checked per release with rpm -q, exactly as ADR 5 pinned the inherited sshd.

corium-agent learns to configure a host interface. This is the first host network interface Corium declares — until now host networking was cloud-init’s alone, and the corium: block spoke only of cluster addressing (pod/service CIDRs, the VRRP VIP). That boundary moves here, a little, and the ADR names it so the next person does not have to rediscover that it moved. detectNodeIP and the HA path both grow an awareness of the overlay interface; a WireGuard overlay carrying a VRRP VIP works only over unicastPeers, since multicast does not cross a point-to-point tunnel, and detectNodeIP must select from the overlay interface rather than the default route. These are real changes to the most delicate code in the tree, and their tests are the price of the feature.

One window this record will not pretend is closed. A peer endpoint given as a hostname has to resolve before the overlay is up, and if DNS itself is only reachable over the overlay that is a cycle the field cannot break. The validation warns and the documentation says to use addresses; the field does not make a hostname endpoint safe, it only makes the failure legible.

Alternatives considered#

Documentation only: a cloud-init recipe, no field. The honest baseline, and rejected on three counts a recipe cannot reach. First, it runs only where cloud-init runs: write_files and runcmd are cloud-init modules, so on an on-premise or bare-metal node with no datasource — configured through /etc/corium/config.yaml or the kernel command line, the paths internal/source exists to serve — the recipe never executes at all, and that is the very population an overlay across sites and providers is for. Second, it cannot make the overlay address the address the kubelet registers, because that choice is made inside corium-agent after cloud-init has run and there is no kubelet-args escape hatch to reach it from a runcmd. Third, it cannot keep the private key out of cleartext instance metadata. Everything else about a recipe works — a write_files config and an enabled wg-quick@.service do bring an interface up before k0s — but those three gaps are what turn a working demo into a node that is unconfigured off-cloud, quietly unreachable from across the overlay, or a cluster whose keys leak with its metadata. That is the same reasoning ADR 3 used to reject documentation-only for spare-disk RAID: the ordering problem is the one that silently breaks a node. A recipe will still be documented as the escape hatch (decision 9), and will still be the right tool for an overlay that is not the cluster transport, on a node that has cloud-init anyway.

A bare node-IP override field plus the recipe. Tempting as the smaller move: ship wireguard-tools, add a general node.address override so the kubelet registers what it is told, and document the interface as a recipe. Rejected because it splits one invariant — “the node registers its overlay address” — across two places the operator keeps in sync by hand, with the agent unable to check that the interface actually carries the address the field names. It also leaves the private key in the recipe’s write_files, unsolved. ADR 3 could decompose cleanly because there was a real seam: spare disks are declarable at first boot, a RAID root is genuinely impossible at first boot. Here there is no such seam — interface bring-up, node addressing and the secret are all first-boot-and-after, agent-adjacent and key-bearing — so a decomposition cuts through the middle of one problem rather than between two.

A NetworkManager keyfile connection instead of wg-quick. NetworkManager manages WireGuard natively and is the one renderer the image already runs, so it could materialise the interface without wg-quick at all. Rejected as the primary path for two reasons: it couples the feature to NM being the renderer forever, where wg-quick is renderer-agnostic and survives a future change of mind; and since wireguard-tools ships present-but-inert regardless — wg is wanted for diagnostics on a node whose overlay is misbehaving — the package it would save is one Corium keeps anyway, so avoiding wg-quick buys little. The schema does not depend on which of the two materialises the interface, so this stays an implementation choice the ADR need not freeze.

Managing the overlay’s keys and mesh from the corium: block. No. Minting and distributing keys, and knowing the whole topology, is fleet management, and Corium provisions nodes rather than managing them. The block describes one node’s peers and no more.