ADR 5 — SSH access over the API

Status: accepted

Context#

ADR 4 made the management API shell-less on purpose. The API reads the state of a node, drives its upgrade, and moves it through its lifecycle. It gives no shell, for two reasons. First, a shell over an immutable system grants root access to change things that will not persist. Second, to the question “how does a node learn whom to obey”, a pairing code over the console is a better answer than a key baked into cloud-init. The API replaced SSH, and ADR 4 said so.

Two facts do not fit well with that position.

First, Corium never removed SSH. It only stopped using it. The base image, quay.io/fedora/fedora-bootc, ships openssh-server. The minimal-plus manifest, which the standard image builds on, pulls it in. The base image keeps sshd enabled through the default systemd preset, and nothing in the Corium overlay turns it off. A node in service already listens on port 22. A closed port does not keep it shut. An empty guest list does: disable_root: true and users: [] in 05-corium.cfg mean that there is no account to log into. The lock is real, but it comes from the absence of a user, not the absence of a daemon.

Second, on day two you sometimes need a shell anyway. For example, a node misbehaves in a way that the four API surfaces were never meant to reach, and the correct tool is to log in and look around. Today, a key can reach a node in only two ways. cloud-init is one, but it closed when the node first booted. cctl reset is the other, but it removes the node from its cluster to change one file. Neither fits “let me in to look.”

Decision 7 is the constraint that shapes any answer. cloud-init is the only authority over users and their SSH keys, so that two systems never race to own an account. Whatever the API does with SSH, it must not become that second authority.

Decision#

apid gets a small surface to manage authorized SSH keys. The surface is narrow enough that decision 7 still holds.

The API manages keys for a user that already exists. It never creates an account, sets a password, chooses a shell, or grants sudo. All of that stays with cloud-init, so the authority over who can log in, and with what privilege, does not change. The API owns key material, and only that.

The API keeps that material in its own file. Corium keeps the keys of a user in /var/lib/corium/ssh/<user>, owned by root. sshd reads them through a drop-in that Corium ships at /etc/ssh/sshd_config.d/10-corium.conf. The drop-in is in /etc because sshd reads drop-ins from no other place. This is the second deliberate exception to the Corium rule about shipping configuration there:

AuthorizedKeysFile .ssh/authorized_keys /var/lib/corium/ssh/%u

The file has mode 0644 in a 0711 directory, under a /var/lib/corium that is also 0711. These modes are necessary, not careless. sshd opens an authorized-keys file as the user who is logging in, not as root, so it silently skips a file that only root can read. The first working version of this feature wrote the file with 0600 in a 0700 directory. The API accepted the key, list returned it, and the journal recorded it as trusted. But sshd refused every login, and no log said why. StrictModes is the rule that actually governs these modes. It refuses a path that other users can write, and other users cannot write any of these paths. The read bit gives nothing away, because a public key is not a secret. For the same reason, list is a readonly call.

The directories are 0711 and not 0755 because permission to pass through a directory is not permission to list it. Nobody needs to list which users have keys. Also, /var/lib/corium holds other things, for example the state directory of the API, which holds the serving key of this node. That directory stays 0700. A user can pass through /var/lib/corium, but cannot open that directory.

The two files never collide. A key that an operator placed by hand, or that cloud-init wrote, is in the file of the user, and the API does not touch it. A key that the API added is in the Corium file, and list and revoke act only there. Each file has one writer.

Three calls, on an enrolled node:

• POST /v1/access/ssh (admin): adds a public key for a named user that already exists. It returns the SHA-256 fingerprint of the key.
• GET /v1/access/ssh (readonly): returns the installed keys, by fingerprint and comment. A public key is not a secret. The ability to read what a node trusts is worth more than the little that hiding it would give.
• DELETE /v1/access/ssh/{fingerprint} (admin): removes one key.

The add and remove calls require admin because of what they grant, not because of what they change. Every other call in this API acts inside the guard rails of the machine. A shell goes outside them. No role is above admin, so this is the most consequential action that apid does. apid logs it loudly, by fingerprint and caller. It never treats the public key as the thing to hide, and it never logs a shell.

cctl reset removes /var/lib/corium/ssh. The keys that the API added go away with the enrolment that authorized them. The account stays, because it belongs to cloud-init, and reset does not change the state of another system.

Consequences#

This is the first deliberate exception to “the API has no shell”. We name it as an exception, and we do not pretend that the line held. The exception is narrow. The API grants a key, not a command. The shell that the key opens belongs to the operating system. The sshd of the OS, its PAM stack, and its SELinux policy control that shell, and apid mediates none of them. apid decides who is trusted. It does not become the thing they talk to.

Corium now depends openly on the presence of an enabled sshd in the base image. Before this decision, Corium inherited this property without stating it. The digest pin of the base image also pins this dependency, and each release checks it again. The dependency also raises a question that this ADR does not settle: will Corium own the decision to run sshd at all, instead of inheriting an enabled daemon? A follow-up will answer it. This decision only makes the inherited daemon usable on purpose.

SELinux does not let sshd read an authorized-keys file outside a home directory unless the file has the correct label. Corium labels these files ssh_home_t, the type that ~/.ssh already has. This labeling is the cost of keeping the Corium keys in their own file. Corium never pays that cost by weakening enforcement.

The method that applies the label needs a precise description, because the obvious method is not available. A file-context rule in the policy (semanage fcontext) is the durable method, and Corium cannot ship it. semanage writes into /var/lib/selinux. The install seeds /var, and an image never updates it. Thus a rule added that way reaches the machines installed with it, and no machine upgraded into it. The same limit stopped Corium from shipping its own SELinux module.

So Corium applies the label with chcon, twice: when it writes a key file, and again every time corium-apid starts. The second pass is necessary, not extra caution. A restorecon -R /var, a policy update, or a filesystem relabel resets these files to the default type for /var. We measured the result on a node, and did not only reason about it. A working login became Permission denied, with no message in any log, on a machine whose API still reported the key as trusted. Because Corium applies the label again at start, a relabeled machine repairs itself on the next boot. The same pass widens the modes that an earlier release wrote. This is what fixes a node that was upgraded into this feature instead of installed with it.

One window stays open, and this record will not pretend otherwise: if a relabel happens while the daemon runs, access breaks until corium-apid starts again. To close the window fully, Corium needs the policy rule that it cannot ship. A watch on the file, or a new label check on every call, would make the gap smaller but not close it. Neither seemed worth the extra machinery for a failure that a service restart fixes.

Alternatives considered#

Model users: and ssh_authorized_keys: in the corium: schema, applied over POST /v1/config. Rejected for two reasons. First, the configuration endpoint refuses requests after a node bootstraps, and this feature exists exactly for that period. Second, users in the schema would make Corium an owner of accounts. That is the collision with decision 7 that this design avoids, not a way around it.

A dedicated Corium-owned break-glass account, which the API creates only when it adds the first key. This is attractive, because it needs nothing from cloud-init. Rejected, because to own an account is to own its password, its shell, and its sudo policy. Decision 7 keeps exactly those things out of the control of Corium. To manage a key for a user that someone else created is a much smaller claim.

An api.insecure-style open door: empty passwords, or SSH with no key at all. No.