cctl

cctl manages Corium nodes. It reads their state, runs their upgrades, and controls which of them obey you. This page tracks main. The current stable release is 0.5.0. If a command is newer than the release that you run, the page gives the version that introduced it.

cctl <command> -h prints the flags of one command. cctl -h prints this list as the binary knows it.

Where it runs, and how to install it#

cctl runs on your machine, never on a node. There is a reason for this split. cctl holds the private key that owns a fleet, and nothing that holds that key belongs in an operating system image. A node ships corium-agent, which does the local work, and corium-apid, which answers cctl over the network.

Install it with mise use -g 'github:Corium-OS/Corium[exe=cctl]@0.5.0'. In zsh, the quotes are necessary. The version is required. You can also download an archive from the release page. downloads covers both methods, and tells you how to verify what you downloaded. From a checkout, run mise run build.

1. What it needs to work#

cctl needs three things, and they must arrive in this order.

An operator CA. The certificate tells a node which authority to trust. The private key next to it signs the certificates that you present. Nodes get the certificate, never the key.

A client certificate, signed by that CA, that carries your role. A node authenticates this certificate on every call.

A node’s fingerprint. A node signs its own serving certificate, because a Corium configuration never carries a private key to sign it with. For this reason, a hostname proves nothing, and the fingerprint proves everything. After the first contact, cctl remembers one fingerprint for each node.

$ cctl pki init
$ cctl pki issue --role admin
$ cctl enroll 192.168.1.51 --code K7QM-93XF --fingerprint SHA256:tQ2f...
$ cctl status 192.168.1.51

2. From nothing to a managed node#

Make the CA#

$ cctl pki init
Created an operator CA in /home/you/.corium

  certificate  /home/you/.corium/operator-ca.pem
  private key  /home/you/.corium/operator-ca.key

The key owns every node that trusts this CA. It never leaves this
machine, and losing it means re-enrolling every node from its console.

Then the command prints the certificate in a form that you can paste into cloud-init. api.operatorCA needs nothing more. pki init refuses to overwrite an existing CA. An overwrite does not only lose a file. It loses every node that pinned the old CA, and each of those nodes then needs a visit to its console. If you really want a new CA, move the directory to a different location first.

Issue yourself a certificate#

$ cctl pki issue --role admin
Signed a client certificate for "you" as corium:admin, valid for 2160h0m0s.

--role is readonly, operator or admin (§4). The default lifetime is ninety days. Client certificates are the credentials that travel on laptops, so they are the ones to keep short. Also, you can reissue one without touching a node.

Claim a node#

A node in maintenance mode prints this message on its console, on its serial port, and in its journal:

Corium node is unenrolled and is not in a cluster.

  address       192.168.1.51:7443
  pairing code  K7QM-93XF
  fingerprint   SHA256:tQ2f...9c1a

  cctl enroll 192.168.1.51:7443 --code K7QM-93XF

Each value has a purpose. The code authenticates you to the node. The fingerprint authenticates the node to you.

You also find the message above the login prompt on the screen of the node. This can be the hypervisor console, the IPMI view, or another console that you have. The message is there because a one-time line scrolls away, and you cannot log in to an unclaimed node anyway. The node removes the message when somebody claims it. So a prompt never shows a code that no longer works.

$ cctl enroll 192.168.1.51 --code K7QM-93XF --fingerprint SHA256:tQ2f...9c1a
Claimed 192.168.1.51:7443.

The node is restarting to require your client certificate, and its
bootstrap is released: it will now join the cluster its cloud-init
configuration describes.

If you omit --fingerprint, cctl shows what answered and asks you to compare it with the console. If nobody is there to answer, cctl refuses. A script that accepts whatever answers checks nothing, but it looks as if it did.

Claiming a node that already knows what it is#

Claiming a node releases a held bootstrap. So if the cloud-init of a node names a role, the node does not wait after you claim it. It immediately starts to become that node. To undo this, you need cctl reset, a drain, and a reboot.

The node tells you this before it acts, and does not decide for you:

$ cctl enroll 192.168.1.51 --code K7QM-93XF --fingerprint SHA256:tQ2f...9c1a

claiming this node releases its bootstrap, and the document it booted with
names a role: it becomes a single node named "ks-stor" in cluster "corium"
immediately, which cannot be undone without cctl reset. Send the configuration
it should build instead, or claim it again saying you meant this.

Claim it anyway? [y/N]

If you answer y, cctl claims the node. The other answer is --config: send the document that the node must build instead. Then the question does not come up, because you said what you want. While the question is open, nothing is claimed and the pairing code stays valid. So either answer costs nothing.

A node that waits to be told what it is, because its document names no role, is claimed without any of this, because claiming it builds nothing. That is the fleet pattern (see configuration §3.2). A prompt on the recommended path only teaches people to ignore it.

In a script, pass --yes. Without a terminal, cctl returns the refusal and does not ask. The refusal names the flag.

If the configuration of a node already names an operator CA, the node never needs this, because it claimed itself at boot. Give cctl its fingerprint one time, from the journal or the instance console log. cctl remembers it:

$ cctl status 192.168.1.51 --fingerprint SHA256:tQ2f...9c1a

Tell a node what it is#

Claiming a node says who owns it. It does not say what the node must become. That comes from the document that the node booted with. For a machine with no cloud-init datasource, there is no such document.

--config fills that gap. A node told to wait for a document holds its bootstrap until the document arrives:

# The whole cloud-config, identical on every machine in the fleet. It names no
# role, which is how a node says it is waiting to be told what it is.
corium:
  api:
    enabled: true

$ cctl enroll 192.168.1.51 --code K7QM-93XF --fingerprint SHA256:tQ2f...9c1a \
    --config controller-01.yaml
Claimed 192.168.1.51:7443.

The node is restarting to require your client certificate, and its
bootstrap is released: it will now join the cluster the configuration
you just sent describes.

The document travels with the claim, not after it. This order is the point of the feature. Claiming a node releases its bootstrap. A document sent a moment later races a machine that already started to become something.

If the node is already claimed, or already bootstrapped, use cctl apply: Applying a configuration.

3. The commands#

Most of these commands take the address of a node, and the port defaults to 7443. Three commands take no address, because they never leave your machine: cctl pki init, cctl pki issue and cctl version.

Every command that reaches a node shares two flags. This table lists them once, so the tables below do not repeat them:

cctl upgrade and cctl ca rotate take --dir but not --fingerprint. Both take a list of nodes, and a single fingerprint has no meaning for a list.

The operator PKI#

These two commands make the credentials that every other command presents. They reach no node and need no role, because roles come from them.

$ cctl pki init
$ cctl pki issue --role admin

cctl pki init:

cctl pki issue:

§2 shows both commands and gives the reasons for them.

Applying a configuration#

cctl apply gives a node the corium: document that it will bootstrap with. If the node already bootstrapped, the command re-applies the part of that document that is safe to change in service. Re-applying on a bootstrapped node is new in 0.2.0. Before 0.2.0, the node refused it.

$ cctl apply 192.168.1.51 --file controller-01.yaml
Applied to 192.168.1.51:7443.

  written to  /etc/corium/config.yaml
  role        controller+worker

The node bootstraps with this the next time its bootstrap runs --
now, if it was holding for one.

The file can be a whole cloud-config with a corium: key, or the block alone. cctl validates it on your machine before it sends it, and the node validates it again before it writes it. So if a document will fail at boot, it is refused while somebody still watches.

Warning. The document is the whole configuration of the node, not a patch. If you leave api: out of it, the node drops everything that api: said. The most important loss is an operatorCA, which is what owns the node. cctl apply tells you when it sees this. Since ADR 11, this no longer turns off the daemon. But a node that bootstrapped and names no owner has nobody left to authenticate.

What the command does depends on the current state of the node:

A new definition of what a node is, while it is in service, makes its configuration and its behavior say two different things. The Corium provisioning model exists to prevent that. See ADR 8.

The identity fields are role, cluster, join, node, network, storage, raid, zfs, wireguard, ha, upgrades, backup and api. The safe subset today is addons, the k0s escape hatch, and kubernetes.version. The project expects this subset to grow. A reconcile regenerates the k0s configuration of the node and restarts the control plane. Then k0s installs or updates the chart, or reads the patched configuration, for example an OIDC extraArgs on the API server. The change is live immediately, not at the next boot. Only a controller does a reconcile. The controllers declare the charts and the cluster configuration of a worker, so a rewrite of the worker’s copy changes nothing that the worker runs. The node enforces the whole rule. cctl does not enforce it, and your role does not change it: an admin certificate does not get past it either.

A change to kubernetes.version is the only apply that takes the node out of service. It replaces the binary under k0s, not a file that k0s reads. The node drains itself, stops k0s, swaps the system extension, clears the staged binaries, starts k0s, and uncordons itself. This is the sequence that k0s documents, in that order. The node downloads the extension before all of this. So if the registry is unreachable, or if the signature does not verify, nothing is lost. If a pod disruption budget refuses the drain, the node cancels the change and does not force it. Expect this apply to take minutes, not seconds.

On a controller, the node takes a k0s backup between the drain and the stop. cctl apply prints its path as snapshot. If the snapshot fails, the node cancels the change and uncordons itself. See upgrades.

There is no automatic rollback if k0s does not come back. The node stops, reports the reason, and stays cordoned. A minor upgrade can migrate state on its way up, and k0s does not support moving back a minor at all. On a node whose control plane already has problems, an unrequested reversal can turn a recoverable problem into one that you cannot recover. See ADR 10.

Corium passes k0s.patch through without changes. So it re-applies under the same contract as at bootstrap: Corium checks only that the result is valid YAML. If a patch rewrites something important, or breaks the cluster, you own the result. A patch cannot change a field that Corium models (role, cluster, and the other fields above). Those fields still need a reset.

Two edge cases are important. Removing an add-on is refused, not applied. k0s installs a chart from its configuration, but it does not uninstall a chart that you remove from it. So the node cannot honestly report the removal as done. Instead, delete the release with kubectl delete chart <name> -n kube-system and remove it from the document. Then it stays out at the next bootstrap. Also, a node bootstrapped by a release older than this feature has no record of the configuration it was built with. So the node refuses its first day-two apply until you re-bootstrap it through cctl reset. A node bootstrapped by a newer release records that baseline itself.

Reading one back#

cctl config is the other half of cctl apply. It prints the corium: document that the node works from. New in 0.6.0. Before 0.6.0, the API accepted the configuration of a node, but did not return it.

$ cctl config 192.168.1.51
corium:
  role: controller+worker
  cluster: lab

This is /var/lib/corium/applied.yaml: what 192.168.1.51:7443 recorded applying.

The document goes to standard output, and everything else goes to standard error. So cctl config node-01 > node-01.yaml and diff <(cctl config node-01) node-01.yaml both do what they look like they do.

The command returns the baseline in /var/lib/corium/applied.yaml, which is what the node acted on. It prefers this file to /etc/corium/config.yaml, which is only what somebody wrote. cctl apply diffs against the same file, and this is intentional. An endpoint that shows a different document from the one that the next apply compares against is worse than no endpoint. If the two files are different, the command tells you:

Warning: /etc/corium/config.yaml has been edited since and does not match. That
edit is not running, and `cctl apply` diffs against this document, not that file.

If a node has no recorded baseline, the command falls back to /etc/corium/config.yaml and labels it. If a node has neither file, the command returns a 409. This happens when the node bootstrapped from cloud-init or from the kernel command line, because the node keeps neither of them.

The command needs corium:admin because of what a document can carry, not because of what reading does. A document can carry an inline join.token, a VRRP authPass, a WireGuard privateKey, and a k0s.patch that Corium only checks for valid YAML. cctl kubeconfig and cctl worker-config already need this role. So the command gives an admin nothing that the admin cannot already create.

Reading#

cctl status is the only command here with a flag of its own:

Use cctl health when something is wrong with your credentials, not with the node. It is the smallest call that proves that a certificate works.

$ cctl status 192.168.1.51
192.168.1.51:7443

  hostname     corium-00a7a34c
  role         single
  cluster      apitest

  os           Fedora Linux 44 (Forty Four)
  kernel       7.2.5-200.fc44.x86_64
  booted       ghcr.io/corium-os/corium:0.1
  digest       sha256:bd67161...

  k0s          v1.36.4+k0s.0
  service      k0scontroller.service (running)
  greenboot    passed
  uptime       11m51s

If the node cannot find the value of a field, the output does not show the field. It does not show a dash or a zero. You usually read this output just before you do something that you cannot undo. A blank is honest, but a placeholder invites a guess. A machine provisioned without a Corium configuration says so clearly, so it does not look broken.

In an incident, the digest is the most important field. A tag says what was asked for. A digest says what booted.

Logs#

$ cctl logs 192.168.1.51 --unit k0scontroller --since 15m --lines 200
23:50:11 info    k0scontroller          Starting kube-router
23:50:12 warn    k0scontroller          node not ready: waiting for CNI

$ cctl logs 192.168.1.51 --unit kernel --lines 20
$ cctl logs 192.168.1.51 --follow

If you omit --unit, the command reads everything that the API knows about. Use this before you know where the problem is. Units come from a fixed list, because an API that passes any unit name to systemctl can start anything on the machine.

Before you give somebody corium:readonly, think about this: the role reads journals, and journals are not sanitized.

Acting on a node#

cctl drain, cctl reboot and cctl shutdown take only the two shared flags.

cordon and drain work on controllers only. Most of a cluster is workers, so this limit applies in the common case, not in a rare one. A node acts only on itself, and only a controller holds cluster admin credentials locally. A worker has the credentials of the kubelet, and these cannot evict a pod.

Draining a worker is a cluster operation, not a node operation. So the tool for it is kubectl, with credentials that this CLI gets for you:

$ cctl kubeconfig <a controller> > kube.yaml
$ kubectl --kubeconfig kube.yaml drain corium-w1 --ignore-daemonsets --delete-emptydir-data
$ kubectl --kubeconfig kube.yaml uncordon corium-w1

The node tells you this when you ask it directly:

$ cctl drain 192.168.1.52
cctl: 409 Conflict: only a controller holds cluster admin credentials, and
this node is not one. Draining a worker is a cluster operation: ...

This limit does not stop cctl upgrade on a worker. The upgrade path drains where it can, and reboots without a drain where it cannot. This is the same behavior as corium-upgrade-apply.service.

A drain that cannot finish is not forced. Usually, a pod disruption budget says that this workload cannot lose a replica now. That is exactly when an override is wrong. The node stays cordoned so that you can decide.

The list of units that you can restart is shorter than the list that you can read. corium-bootstrap.service is readable, but intentionally not restartable. If it runs again on a node that already joined a cluster, it destroys data. No certificate can ask for it.

$ cctl restart 192.168.1.51 --unit corium-bootstrap
cctl: 403 Forbidden: corium-bootstrap.service: this unit is not one the API
will restart (first-boot configuration; runs once)

reboot and shutdown reply before they act. Without a reply, “the node refused” and “the node obeyed” look the same: both look like a connection that died.

A dashboard in the browser#

cctl ui serves a page that shows every node at once. On that page, restart, cordon and drain are buttons. It runs on your machine. The server binds to loopback, holds your client certificate, and talks to nodes over the same mutual-TLS API that every other command uses. When you open a browser, no node gets a new port, password, or session. For the same reason, ADR 4 put the management API on the node and the view of the fleet nowhere.

$ cctl ui
cctl ui: 3 node(s)

  http://127.0.0.1:7500/?token=hizbucmhfayrnhm2wiiyevbkefdyxf6u

Ctrl-C to stop.

If you name no node, the page shows every node for which ~/.corium/config.yaml already holds a fingerprint (§5). This is the set of nodes that this directory can reach without more information. It is not an inventory. A node that somebody else enrolled does not appear until you name it one time.

$ cctl ui 192.168.1.51 192.168.1.52
$ cctl ui 192.168.1.51 --fingerprint SHA256:2Qn... # a node claimed from cloud-init

What it shows. There is one card for each node. The card shows the hostname, role, cluster, and k0s version, and whether its service runs. It also shows greenboot, uptime, and a mark when an image is staged and waits for a reboot. If a node does not answer, it keeps its card, and the card says why. The usual question is which machine is down. When you open a card, it shows the full cctl status report, the service list, and a journal window. Fields that the node cannot find are not shown, exactly as in the terminal.

The URL contains the open node. So a reload returns to that node, not to the list, and the back button of the browser works as you expect. The page follows the light or dark setting of your system.

What it does. Restart a unit, cordon, uncordon, drain. That is the corium:operator half of the API, and nothing more. The dashboard gets no kubeconfig and no join token. It cannot reboot, shut down, reset, apply a configuration, or upgrade anything. Those actions cannot be undone, or they are rollouts across several nodes in an order that is important. A button is the wrong tool for both. If your certificate cannot do an action, its button is disabled, and the dashboard does not try it.

The token is the credential. Anyone who reaches the port and has the token acts as you. The server prints the token one time, moves it into an HttpOnly cookie at the first load, and keeps it only while the process runs. The server also refuses a Host header that is not its own loopback address. This stops a hostile page that resolves a name to 127.0.0.1 from talking to the server. The server also refuses a request that the browser labels cross-site. If you give --listen a real interface, all of this is open on your network. The warning that the command prints says so.

SSH access#

The API gives no shell of its own (§7). But it can trust an SSH key for a user that already exists on the node. Then you can get a shell in the usual way when the features above are not enough. The API never creates the account. Cloud-init owns the user, its password, and its shell. Corium manages only its own key file, next to the user’s own ~/.ssh.

Warning. add needs admin, because a trusted key grants a shell. A shell is the one thing that goes outside the guard rails that the rest of the API keeps. The node logs the key by fingerprint. Listing needs readonly, because a public key is not a secret.

$ cctl access ssh add 192.168.1.51 --user core --key-file ~/.ssh/id_ed25519.pub
trusted SHA256:PZ8s… (ssh-ed25519) for core

$ cctl access ssh list 192.168.1.51
  core             ssh-ed25519     SHA256:PZ8s…  you@laptop

$ cctl access ssh revoke 192.168.1.51 --key-fingerprint SHA256:PZ8s…
revoked SHA256:PZ8s… for core

The key comes from --key-file, from --key inline, or from standard input. So cctl access ssh add <node> --user core < key.pub works, and piping ssh-add -L also works.

The user must exist first. A key for an account that cloud-init never created gets a 409, and the message tells you what to do:

$ cctl access ssh add 192.168.1.51 --user ghost --key-file key.pub
cctl: 409 Conflict: "ghost": no such user on this node; create it with
cloud-init before adding a key for it

Corium keeps these keys in its own file. So list and revoke never touch a key that you added by hand to the user’s ~/.ssh/authorized_keys. cctl reset removes every key that the API trusted. See ADR 5.

Upgrades#

Warning. cctl upgrade takes one node at a time and stops at the first node that does not come back. A rollout that continues past a broken machine turns one outage into a cluster-wide outage. The error tells you how many nodes were upgraded, because somebody must decide what to do with a half-upgraded cluster.

$ cctl upgrade node-1 node-2 node-3 --image ghcr.io/corium-os/corium:0.5
Checking 3 nodes can move to ghcr.io/corium-os/corium:0.5
  node-1:7443: on ghcr.io/corium-os/corium:0.4
  ...

[1/3] node-1:7443
        staged sha256:bbbb2222
        draining and rebooting.......
        up on sha256:bbbb2222
[2/3] node-2:7443
...

Every node is checked before the first one reboots. Each node must be reachable and safe to lose: bootstrapped, with k0s running, and with a last boot that greenboot did not mark as bad. Each node must be in the same cluster as the others, and listed one time. Its signing policy must accept the image. One refusal stops the rollout, and nothing changes. --dry-run also checks etcd, then stops. If a node is older than this check, cctl reports it, and the node checks the image itself when it stages it.

Just before each node, cctl asks for its health again. After each node, it makes sure that the node came back on the digest it was sent to, not only that it answers.

Around each controller, cctl also checks etcd: before it takes the controller down, and again after the controller is back, for up to --settle. See etcd around controllers.

A node refuses an image if its own signing policy does not require a signature for that image:

$ cctl upgrade node-1 --image quay.io/fedora-ostree-desktops/silverblue:44
Checking 1 nodes can move to quay.io/fedora-ostree-desktops/silverblue:44
  node-1:7443: 403 Forbidden: the node's signing policy does not require a
  signature for this image: quay.io/fedora-ostree-desktops/silverblue:44
cctl: 1 of 1 nodes cannot move to quay.io/fedora-ostree-desktops/silverblue:44;
nothing was changed

This check stops a typo from rebasing a Kubernetes node onto a desktop. It is not a label check, because anybody can label an image “Corium”. It uses /etc/containers/policy.json. The image ships this file with a requirement for a cosign signature on the Corium repository. To run your own derived images, add your repository and key to that file. That is also how you state that you trust them.

$ cctl rollback 192.168.1.51

rollback marks the previous image as the next to boot, and intentionally does not reboot. You use it when something already went wrong, so the reboot stays yours to schedule. A node that booted only one image has no previous image to go back to, and says so. The command takes only the two shared flags.

Kubernetes version rollouts#

cctl upgrade moves nodes to another OS image. cctl k8s upgrade moves a cluster to another Kubernetes version. ADR 10 makes this a separate axis.

$ cctl k8s upgrade --version v1.36.4+k0s.1     --controllers node-1,node-2,node-3 --workers node-4,node-5
Checking 5 nodes can move to k0s v1.36.4+k0s.1
  node-1:7443 (controller): on k0s v1.35.8+k0s.1
  ...

[1/5] node-1:7443 (controller)
        on k0s v1.35.8+k0s.1, moving to v1.36.4+k0s.1
        snapshot /var/lib/corium/backups/corium-backup-20261101T120000Z.tar.gz
        up on k0s v1.36.4+k0s.1
[2/5] node-2:7443 (controller)
...

Controllers and workers are named separately, and the order is not a preference. k0s requires that you upgrade controllers before workers. A worker must never be newer than the controllers that it talks to. When you name them, you state the shape of the cluster before anything moves. So the rollout does not find a mistake in the address list halfway through.

Every node is checked before the first one moves. Each node must be reachable and healthy, in the stage that its role belongs to, and in the same cluster as the others. Each node must accept the version against its own window and skew rules. One refusal stops the rollout, and nothing changes. --dry-run stops there in all cases. If a node is older than this check, cctl reports it, and the node checks itself when it moves.

A stopped rollout resumes when you run the same command again. Nodes that are already on the target version are skipped. The remaining nodes must all start from the same minor. A mix is refused, and the error lists the nodes by minor. For example, a cluster left halfway by an earlier rollout to another version is a mix.

etcd is checked before and after each controller, as cctl upgrade does. The etcd member of a moved controller gets up to five minutes to catch up. See etcd around controllers.

Each node downloads the extension before it goes out of service. So an unreachable registry, or a signature that does not verify, costs nothing. Then the node drains, stops k0s, swaps the extension, and comes back, without a reboot. If a pod disruption budget refuses the drain, the node cancels the change and does not force it. The node then checks that k0s really started from the new binary. If it did not, the change fails and the node stays cordoned.

A controller takes a k0s backup between the drain and the stop. The rollout prints the archive’s path on the node’s snapshot line. If the snapshot fails, the node cancels the change and uncordons itself, and the rollout stops. See upgrades.

A stopped rollout is a safe state. Like cctl upgrade, this command stops at the first node that fails. But unlike cctl upgrade, its half-finished state is one that k0s explicitly supports: controllers one minor ahead of workers. You do not need to hurry to finish. Remember this when you decide whether to continue.

There is no automatic rollback if k0s does not come back on a node. The node stays cordoned and says why. k0s does not support downgrading a minor, so an unrequested reversal can make a recoverable problem permanent.

--json prints one report when the rollout ends, whether it succeeded or not. The exit status still tells you which. A controller’s entry has backup: the snapshot it took before the stop.

$ cctl k8s upgrade --version v1.36.4+k0s.1 --controllers node-1 --workers node-2 --json 2>/dev/null
{
  "version": "v1.36.4+k0s.1",
  "dryRun": false,
  "nodes": [
    {"address": "node-1:7443", "stage": "controller", "from": "v1.35.8+k0s.1", "to": "v1.36.4+k0s.1", "result": "moved", "backup": "/var/lib/corium/backups/corium-backup-20261101T120000Z.tar.gz"},
    {"address": "node-2:7443", "stage": "worker", "from": "v1.35.8+k0s.1", "to": "v1.36.4+k0s.1", "result": "failed", "error": "..."}
  ]
}

If a controller moved but left etcd short of a member, the report shows it as moved, with the error that stopped the rollout.

etcd around controllers#

Both rollouts refuse to take a controller out while etcd is short of a member. They also stop after a controller whose member does not come back healthy:

[1/5] node-1:7443 (controller)
        etcd: 3 of 3 members healthy
        on k0s v1.35.8+k0s.1, moving to v1.36.4+k0s.1
        up on k0s v1.36.4+k0s.1
        etcd: 3 of 3 members healthy

No single node answers for the cluster. Each controller reports, on GET /v1/etcd, the health of its own member and the member list that it sees. The health comes from etcd’s /health, which needs a quorum to pass. cctl compares the answers and stops when:

• A member is unhealthy, or cctl cannot ask a controller.
• The controllers do not agree on the member list.
• A member is on none of the given controllers. A dead machine that is still listed counts against the quorum. List every controller, or remove the dead machine with k0s etcd leave.

A single node runs kine, so cctl skips it. cctl also skips a controller on SQLite or on an external etcd. If a node is older than this check, cctl reports it and does not block. The member of that node is then not checked.

Getting a kubeconfig#

Warning. This command needs admin, and it is more powerful than every other command here. The other commands act on one machine. This one gives away a cluster, and nothing in this API can take it back. The node records the event in its journal.

$ cctl kubeconfig 192.168.1.51 > ~/.kube/corium.yaml
$ cctl kubeconfig 192.168.1.51 --output ~/.kube/corium.yaml

By default, the kubeconfig goes to standard output, and intentionally not to ~/.kube/config. A merge into the existing contexts of a user is a decision that you cannot undo. --output writes a file at 0600, and refuses to replace a file without --force.

On an HA control plane, the server address is the virtual IP of the cluster. Otherwise, it is the address that you used to reach the node. A kubeconfig that points to one controller stops working when that controller stops. That is why the virtual IP wins when there is one.

--server overrides the address. Use it for a load balancer in front of the control plane, for a name instead of an address, or for a port that is not 6443:

$ cctl kubeconfig 192.168.1.51 --server k8s.example.com
$ cctl kubeconfig 192.168.1.51 --server 192.168.0.200:8443

A bare address gets https:// and :6443. A full URL stays as it is.

Only a controller can answer. A worker holds kubelet credentials, which are not administrator credentials. The worker says so, and does not give you something that looks correct but is not.

Generating a worker’s configuration#

New in 0.3.0.

Warning. The token is inline in clear text, by design, so the printed block is a secret. The block goes alone to standard output, so > worker.yaml writes only YAML. The reminder goes to standard error.

Warning. This command needs admin, and only a controller can answer. The token comes from k0s, which a worker does not run. A worker says so, and does not fail with an unclear error.

$ cctl worker-config 192.168.1.51 --name w-1 --label corium.dev/pool=general > worker.yaml
Minted a worker join token on 192.168.1.51:7443, valid 1h. It is inline in the
block above -- treat the output as a secret, and mint a fresh one once it expires.

The command asks a controller to create a new worker join token. The controller runs k0s token create for you. Then the command prints the corium: block for a new worker to standard output:

corium:
  role: worker
  node:
    name: w-1
    labels:
      corium.dev/pool: general
  join:
    token: <a freshly minted worker token, inline>

The block holds only the part for the worker, to paste into a cloud-config. You add the users, the SSH keys, and anything else yourself.

Handing a node to somebody else#

$ cctl pki init --dir ~/.corium-2027 --name "corium operators 2027"
$ cctl ca rotate node-1 node-2 --dir ~/.corium --to ~/.corium-2027
node-1:7443 now obeys the CA in /home/you/.corium-2027

cctl creates a certificate under the new CA and sends it to each node as proof. The node verifies that the certificate chains to the CA that it must obey, and refuses otherwise. This protects against a failure that you cannot recover from over the network. If you rotate to a CA that you cannot issue certificates under, the node will only accept somebody else.

Rotation changes neither cluster membership nor node identity. The fingerprint that you pinned stays valid. Only the person who can manage the node changes.

If the key of the CA is lost, there is nothing to rotate with. The way back is the console: run corium-agent api set-ca --file operator-ca.pem as root on the node itself (§6).

Erasing a node#

$ cctl reset 192.168.1.51
cctl: this erases corium-00a7a34c (192.168.1.51:7443): it leaves its cluster,
forgets its owner and reboots unclaimed. Re-run with --confirm
corium-00a7a34c to mean it

$ cctl reset 192.168.1.51 --confirm corium-00a7a34c

You must send the name of the node back to it. An address in a shell history does not protect well against a reset of the wrong machine. The name is checked twice: cctl compares it with what the node calls itself, and the node checks it too.

The order is fixed. First, the node drains, as a best effort. The node that somebody wants to reset is often one whose cluster is already gone. Then k0s reset takes the node out of the cluster and wipes /var/lib/k0s. Then the node removes the bootstrap marker. Only then does the node forget its owner, and last of all it reboots. The opposite order can leave a cluster member that nobody owns.

The serving identity is also removed, so the fingerprint remembered here stops matching. This is intentional. If a machine goes to a new owner with the certificate that its previous owner pinned, the tools of that previous owner still accept it.

Two things to know:

• What the node comes back as depends on its configuration. A node with api.operatorCA re-claims itself and re-bootstraps, which is a real reprovision. A node in maintenance mode comes back unclaimed, with a new pairing code.
• The reboot takes a staged image if one waits. If that is important, look at cctl status first.

Version#

$ cctl version
cctl 0.5.0 (a2575f3)

The build writes the version and the short commit into the binary at link time. A release binary reports its tag without the leading v. A binary built from a checkout reports the output of git describe --tags --always --dirty. So a build from an uncommitted tree shows itself as dirty. The command takes no flags, reads no operator directory, and reaches no node. So it is the only command that works before you set up anything.

4. Roles#

The role is in the organization field of the client certificate, and the node checks it on every call. Every route names the lowest role that can use it. This table is the authoritative one. The role columns in §3 repeat it.

corium:readonly is the form that the certificate carries and that a node checks for. cctl pki issue --role and cctl ca rotate --role take the short form (readonly, operator, admin) and write the full name.

Three commands are outside this table. cctl enroll runs before any trust exists, so the pairing code replaces a role. cctl pki and cctl version reach no node at all.

cctl upgrade uses two roles. Pulling an image changes nothing else, so it is operator work. Applying the image takes the node out of service. So an operator can stage an upgrade but cannot finish one. cctl upgrade needs admin to run to the end.

A certificate signed by the operator CA but with no recognized role is authenticated, but not authorized. It gets a 403 that tells you how to reissue it. A certificate signed without a role does not get every role.

5. Your directory#

The default is ~/.corium. Use --dir for a different directory. The directory is kept at 0700, whatever the modes of the files inside it. If its mode was wider, cctl narrows it, because the directory holds the key that owns a fleet.

config.yaml holds fingerprints because a node signs its own certificate. If you pass --fingerprint explicitly, cctl records it. A node claimed from cloud-init needs this one time.

6. What runs on the node instead#

Two tools are intentionally not in cctl, because they are local by nature.

corium-agent api set-ca --file <cert> replaces the operator CA that a node obeys. It is the way back when the key that owns a fleet is lost. It opens no port and accepts no request. Root on the machine already owns the machine, so the command grants nothing new. The node keeps its cluster membership, which makes this a recovery and not a reset. The command refuses a node that nobody claimed, because a CA installed there is an enrolment that skips the pairing code.

corium-agent validate <file> parses and checks a configuration offline, without applying it. It uses the same code that the node runs at boot. So it gives the same answer on your workstation as on the machine.

7. What it will not do#

The last row is the design, not an omission. A node knows only about itself. The sequencing that needs to know about the other nodes lives here, not on a node. See ADR 4.

SSH access does not contradict this. cctl access ssh (§3) trusts a key for a user that already exists, but the API still runs no command itself. The shell that follows comes from sshd in the operating system. The API decides only whose key sshd reads. See ADR 5.

8. Reading a refusal#

The status codes tell you where to look from the code alone.

The exit codes that cctl itself returns to the shell are a much shorter list. This is important if you use cctl in a script:

There is no exit code for each cause. A 403 from the node, an address that never answered, and a missing client certificate all exit 1. A script that must tell them apart must read the message, or the status code in it.

A handshake that fails before all of those is a separate case:

$ cctl status 192.168.1.51
cctl: 192.168.1.51:7443 does not accept your certificate: it is signed by a CA
this node no longer trusts.

The raw TLS alert for this is certificate required, which reads as if the client sent no certificate. Usually that is true. Go sends no certificate at all when its certificate was signed by a CA that the server did not name as acceptable. So the real cause is a CA that somebody rotated away from this directory. Use the directory that the CA was rotated to, or recover the node from its console with corium-agent api set-ca.

Read a 409 carefully, because it is easy to read as a fault, and usually it is not one. A node with nothing staged, a node with no earlier image to roll back to, and a worker with no credentials to evict a pod are all normal states, not problems.