ADR 5 — SSH access over the API

Status: accepted

Context#

ADR 4 made the management API deliberately shell-less. It reads a node’s state, drives its upgrade, and moves it through its lifecycle — and it hands out no shell, because a shell over an immutable system grants root to change things that will not persist, and because a pairing code over the console is a better answer to “how does a node learn whom to obey” than a key baked into cloud-init. SSH was the thing the API replaced, and ADR 4 named it as such.

Two facts sit awkwardly against that posture.

The first is that Corium never removed SSH; it only stopped using it. The base image, quay.io/fedora/fedora-bootc, ships openssh-server — it is pulled in by the minimal-plus manifest the standard image builds on — and leaves sshd enabled through the default systemd preset, and nothing in Corium’s overlay turns it off. A node in service is already listening on 22. What keeps it shut is not a closed port but an empty guest list: disable_root: true and users: [] in 05-corium.cfg mean there is no account to log into. The lock is real, but it is the absence of a user, not the absence of a daemon.

The second is that day two sometimes needs a shell anyway — a node misbehaving in a way the four API surfaces were never meant to reach, where the honest tool is a login and a look around. Today the only ways to place a key on a node are cloud-init, which closed the moment the node first booted, and cctl reset, which throws the node out of its cluster to change one file. Neither fits “let me in to look.”

Decision 7 is the constraint that shapes any answer: cloud-init is the only authority over users and their SSH keys, precisely so that two systems never race to own an account. Anything the API does with SSH has to avoid becoming that second authority.

Decision#

apid grows a small surface for managing authorized SSH keys, drawn narrowly enough that decision 7 still holds.

The API manages keys for a user that already exists. It never creates an account, sets a password, chooses a shell, or grants sudo — all of that stays cloud-init’s, so the authority over who may log in, and with what privilege, is unchanged. What the API owns is key material, and only that.

It owns that material in its own file. Corium’s keys for a user live in /var/lib/corium/ssh/<user>, root-owned, and sshd reads them through a drop-in shipped at /etc/ssh/sshd_config.d/10-corium.conf — /etc because that is the only place sshd reads drop-ins from, which makes this the second deliberate exception to Corium’s rule about shipping configuration there:

AuthorizedKeysFile .ssh/authorized_keys /var/lib/corium/ssh/%u

The file is 0644 in a 0711 directory, under a /var/lib/corium that is itself 0711, and that is load-bearing rather than careless. sshd opens an authorized-keys file as the user who is logging in, not as root, so a root-only file is a file it silently skips. The first working version of this feature was written 0600 in a 0700 directory: the key was accepted, list returned it, the journal recorded it as trusted, and every login was refused with nothing said anywhere about why. StrictModes — the rule that actually governs these modes — refuses a path others can write, which none of this is, and nothing is given away by the read bit, since a public key is not a secret. That is the same reason list is a readonly call.

0711 rather than 0755 because traversal is not listing: nobody needs to enumerate which users have keys, and /var/lib/corium holds other things, including the API’s own state directory with this node’s serving key. That one stays 0700, and walking past a door is not opening it.

The two never collide. A key an operator placed by hand, or one cloud-init wrote, sits in the user’s file, and the API does not touch it; a key the API added sits in Corium’s file, and list and revoke act only there. There is one writer per file.

Three calls, on an enrolled node:

• POST /v1/access/ssh (admin) — add a public key for a named, existing user; returns its SHA-256 fingerprint.
• GET /v1/access/ssh (readonly) — the installed keys, by fingerprint and comment. A public key is not a secret, and being able to read what a node trusts is worth more than the little that hiding it would buy.
• DELETE /v1/access/ssh/{fingerprint} (admin) — remove one.

admin, and not for what it changes but for what it grants. Every other call in this API acts within the machine’s guard rails; a shell steps outside them. There is no role above admin, so this is the most consequential thing apid will do, and it is logged loudly — by fingerprint and caller, never treating the public key as the thing to hide, and never logging a shell.

cctl reset removes /var/lib/corium/ssh. The keys the API added leave with the enrolment that authorised them. The account does not: it is cloud-init’s, and reset does not reach into another system’s state.

Consequences#

This is the first deliberate crack in “the API has no shell”, and it is worth naming as one rather than pretending the line held. It is a narrow crack. The API grants a key, not a command; the shell it opens is the operating system’s, gated by the OS’s sshd, its PAM stack and its SELinux policy, none of which apid mediates. apid decides who is trusted; it does not become the thing they talk to.

Corium now depends, in the open, on sshd being present and enabled in the base image — a property it used to inherit silently. That dependency is pinned with the base image by digest and re-checked per release. It also raises a question this ADR does not settle: whether Corium should own the decision to run sshd at all, rather than inheriting an enabled one. That is left to a follow-up; this decision only makes the inherited daemon usable on purpose.

SELinux will not let sshd read an authorized-keys file outside a home directory unless it is labelled for it, and Corium labels these ssh_home_t — the type ~/.ssh already carries. That labelling is the cost of keeping Corium’s keys in their own file, and it is never paid by weakening enforcement.

How it is applied is worth stating precisely, because the obvious answer is not available. A file-context rule in the policy — semanage fcontext — would be the durable way to do it, and it cannot be shipped: semanage writes into /var/lib/selinux, and /var is seeded at install and never updated by an image, so a rule added that way would reach machines installed with it and no machine upgraded into it. That is the same wall that stopped Corium shipping an SELinux module of its own.

So the label is applied with chcon, twice: when a key file is written, and again every time corium-apid starts. The second is not belt and braces. A restorecon -R /var, a policy update or a filesystem relabel resets these files to the default type for /var, and the consequence was measured on a node rather than reasoned about — a working login became Permission denied, with no message in any log, on a machine whose API still reported the key as trusted. Reapplying at start means a machine that has been relabelled repairs itself on the next boot. The same pass widens modes an earlier release wrote, which is what carries a node upgraded into this feature rather than installed with it.

One window stays open, and this record will not pretend otherwise: a relabel while the daemon is running breaks access until corium-apid is next started. Closing it properly needs the policy rule that cannot be shipped. Watching the file, or re-checking the label on every call, would narrow the gap and not close it, and neither seemed worth the machinery for a failure whose fix is a service restart.

Alternatives considered#

Model users: and ssh_authorized_keys: in the corium: schema, applied over POST /v1/config. Rejected on two counts. The config endpoint is refused once a node has bootstrapped — which is exactly the window this feature exists to serve — and modelling users in the schema would make Corium an owner of accounts, which is the collision with decision 7 this design is built to avoid, not a route through it.

A dedicated Corium-owned break-glass account, created lazily by the API on the first key. Tempting, because it needs nothing from cloud-init. Rejected because owning an account is owning its password, its shell and its sudo policy, and that is the whole of what decision 7 keeps out of Corium’s hands. Managing a key for a user someone else created is a much smaller claim.

An api.insecure-style open door: empty passwords, or SSH with no key at all. No.