This page lists what Corium models, what it passes through, and what it does not do on purpose.

Read this page before you commit to Corium, to find out if Corium will get in your way. The short answer: Corium models the settings that most clusters need, and everything else in k0s stays reachable through a passthrough patch. Corium does not prevent you from configuring any k0s feature. It only gives a shorter name to some of them and not to others.

Each description of k0s behavior on this page links to the official k0s documentation. Corium does not fork or patch k0s. If the two disagree, k0s is right and this page is out of date.


Three levels of support

LevelMeaning
ModelledA first-class field in the corium: schema. Corium validates it, gives it a default, and documents it in the configuration reference
PassthroughNot modelled, but reachable verbatim through k0s.patch. Corium does not validate it or change it
Out of scopeNot supported, on purpose. Each one has a reason below

Passthrough is not a lower tier for forgotten settings. See why a setting ends up there.


Modelled

Cluster shape

FeatureFieldNotes
Single-node clusterrole: singleImplies SQLite storage. You cannot add nodes later
Control plane onlyrole: controllerRuns no workloads
Control plane plus workloadsrole: controller+workerCorium passes --no-taints, so the node runs workloads
Workerrole: workerRequires a join token
Cluster namecluster.nameAppears in the generated kubeconfig contexts
API endpointcluster.endpointBecomes spec.api.externalAddress. Corium also adds it to the certificate
Extra certificate namescluster.subjectAltNamesAppended to spec.api.sans

Networking

FeatureFieldNotes
Pod address rangenetwork.podCIDRDefault 10.244.0.0/16, matching k0s
Service address rangenetwork.serviceCIDRDefault 10.96.0.0/12, matching k0s
kube-routernetwork.cni: kuberouterThe k0s default: networking, policy, and service proxy in one component
Caliconetwork.cni: calico
Bring your own CNInetwork.cni: customk0s deploys nothing. The node stays NotReady until you install a CNI. Worked example with Cilium

Storage

FeatureFieldNotes
Embedded etcdstorage.type: etcdDefault for every multi-node role
SQLitestorage.type: sqliteRendered as the k0s kine. For a single controller only. Corium rejects it with role: controller

High availability

FeatureFieldNotes
Virtual IP for the control planeha.enabled, ha.virtualIPk0s control plane load balancing with keepalived. No external load balancer needed
VRRP groupha.virtualRouterIDMust be unique in the broadcast domain
VRRP passwordha.authPass, ha.authPassFromEight characters at most, because keepalived cuts longer passwords without a warning
Interfaceha.interfaceDefaults to the interface that holds the default route
Non-multicast networksha.unicastPeersRequired on most clouds

Nodes

FeatureFieldNotes
Node namenode.nameDerived from the machine ID when unset
Labelsnode.labelsCorium sorts them before they reach the command line, so identical input gives an identical command
Taintsnode.taintskey, value, effect
Node addresswireguard[].nodeAddressSet automatically with HA (without the virtual IP), or pinned to a WireGuard overlay interface

Joining

FeatureFieldNotes
Inline tokenjoin.tokenOr minted and rendered for you by cctl worker-config
Token from a URL or filejoin.tokenFromHTTPS only
Waiting for a token to appearjoin.tokenFrom.waitForEvery node can start at the same time instead of one after another
Bearer auth for that URLjoin.tokenFrom.authFile

Add-ons

FeatureFieldNotes
Helm charts at bootstrapaddons[]Rendered into spec.extensions.helm. No Helm binary and no in-cluster operator
Chart repositoriesaddons[].repository
Chart valuesaddons[].valuesConverted to the YAML string k0s expects

Bundled manifests

FeatureFieldNotes
Plain Kubernetes YAML at bootstrapmanifests[]Written to /var/lib/k0s/manifests/<stack>/ before k0s starts. Use it for objects that have no chart. See Manifest Deployer
Several files, several documentsmanifests[].filesOne file can hold documents separated by ---, as kubectl accepts them
Checked offlinemanifests[].files[].contentcorium-agent validate parses it as YAML. Corium has no schema for a Kubernetes object and does not claim to have one
.yml refusedn/aThe deployer reads only the .yaml extension and skips a .yml file without a warning. Corium rejects .yml instead
Reconciling and pruningn/aDone by k0s, not by Corium. k0s watches each stack and prunes the resources of a file that you remove

Upgrades

FeatureFieldNotes
Unattended stagingupgrades.automatic: downloadStages a new image, and never reboots on its own
Unattended rebootupgrades.automatic: applyDrains the node, reboots, and uncordons. If a PDB refuses the drain, the upgrade is cancelled
Check scheduleupgrades.schedulesystemd OnCalendar, default daily
Version laddern/a0.5.0, 0.5 and latest per release. No major rung below 1.0, because a 0.x minor can break things. See upgrades
Automatic rollback on failuren/agreenboot health check. A node that cannot run k0s returns to its previous image

Backups

FeatureFieldNotes
Scheduled control plane snapshotbackup.enabledA systemd timer that runs k0s backup: etcd or kine, the PKI, k0s.yaml, manifests, and Helm configuration. For controllers only. Corium rejects it on a worker
Schedulebackup.schedulesystemd OnCalendar, default daily
Target directorybackup.pathDefault /var/lib/corium/backups. Corium writes archives with mode 0600 in a 0700 directory
Retentionbackup.keepDefault 7. Deletes only the archives that Corium named, never a glob
Restoren/aManual. An operator decides to run k0s restore on a stopped node, not a timer. See reference §3.18
Workload datan/aOut of scope. A k0s backup holds the control plane, not PersistentVolumes

Kubernetes version

FeatureFieldNotes
Pin a Kubernetes versionkubernetes.versionIndependent of the OS image. A signed system extension overlays /usr/bin/k0s. If empty, the node runs the version that the image ships. See ADR 10
Pull from an internal registrykubernetes.mirrorFor a site that cannot reach the default registry. A mirror changes where an artefact comes from. The artefact must still be signed
Supported windown/aThree Kubernetes minors per image, listed in /usr/lib/corium/k0s.window. To get a different minor, upgrade the OS
Skew enforcementn/aOne minor at a time, and no move back to an older minor. These are the k0s rules. Corium refuses a change that breaks them, names the rule, and stops nothing
Rolling a version backn/aNot free, as an OS rollback is. k0s does not support a downgrade of a minor, so a revert is a second forward move

Disks

FeatureFieldNotes
Software RAID on spare disksraid[]Levels 0, 1, 5, 6, 10. Created before k0s starts
Hot sparesraid[].sparesAdded automatically when a member fails
Filesystemraid[].filesystemext4 (default), xfs, or none for a raw device
Mount and persistraid[].mountPointWritten to /etc/fstab by UUID
Refuses to destroy dataraid[].wipeOff by default. A device that holds data stops the bootstrap
Root filesystem on RAIDn/aWorks, but only at install time. It needs a hand-written Kickstart, a second ESP, and an fstab edit. See software RAID
ZFS pools on data diskszfs[]Mirror or raidz, created before k0s. Needs the ZFS image variant. See ZFS on data disks
ZFS datasetszfs[].datasetsPer-dataset mount points and properties (compression, recordsize, quota)
ZFS refuses to destroy datazfs[].wipeOff by default. A device in a pool cannot also be in a raid[] array
Root filesystem on ZFSn/aOut of scope. It needs the module in the initramfs, and bootc has no declarative way to put it there. See ADR 7

Overlay networking

FeatureFieldNotes
Host WireGuard interfacewireguard[]Started before k0s, so that a cluster can run over an encrypted overlay between hosts. See ADR 6
Overlay as the node addresswireguard[].nodeAddressThe kubelet registers this address, not the physical NIC. This field makes the overlay the transport for the cluster
Peerswireguard[].peersPublic key, endpoint, allowed IPs, keepalive, and an optional preshared key
Private key as a secretwireguard[].privateKeyFromResolved at first boot, written with mode 0600, and never logged, as for a join token
k0s waits for the overlayn/aThe k0s service requires the interface. If the overlay fails, the node does not half-join

Not the same as a CNI’s encryption

This is a host concern. It is different from the pod-traffic encryption that a CNI gives. The overlay lets hosts on different sites or providers reach each other at all. wireguard-tools is in the image but does nothing until you declare an interface.

Management API

Off unless you ask for it. The decision is ADR 4. The commands are in cctl.

Schema fields

FeatureFieldNotes
The API itselfapi.enabledOn by default. enabled: false is the only way to run no daemon and bind no port. A bootstrapped node that names no operator CA also binds no port
Operator CA, inlineapi.operatorCAThe certificate of the CA that signs operator client certificates. It is public, so it is safe in clear text in cloud-init
Operator CA, resolvedapi.operatorCAFromThe same SecretSource as join.tokenFrom, waitFor included
Maintenance modeapi.enabled: true, no CAThe node holds its bootstrap and prints a pairing code on the console until an operator claims it
Open enrolmentapi.insecureRemoves the pairing code. The first client that reaches the node claims it

api.insecure is for test benches and controlled provisioning networks. The node shows this on its console, and records that its claim was not authenticated.

A node in maintenance mode is not a cluster member. It validates its configuration and waits. It does not join first and then wait for a claim. Enrolment carries no configuration. It sends a CA certificate and nothing else. The role and the join token of the node still come from cloud-init.

How the daemon works

PropertyNotes
TransportJSON over HTTP and mutual TLS on 7443. It uses only the standard library, so no dependency ships to a node for it
RolesCarried in the organization field of the client certificate: corium:readonly, corium:operator, corium:admin. Every route names the lowest role that can call it
Image validationA node refuses an image that its own signing policy accepts without a signature. This is not a label check, because anyone can forge a label
SELinuxThe daemon runs unconfined, as the agent does. /var/lib/corium/api has its own type, so that a confined domain has an anchor. The parent keeps var_lib_t because greenboot and systemd read it

What it can be asked to do

These are commands, not schema fields. The syntax, flags, and output are in cctl.

CommandWhat it does
cctl statusRole, cluster, booted and staged image digests, kernel, k0s version and service, greenboot result, and uptime. Read-only
cctl servicesLists the units that the API knows about. Units come from a fixed list. The caller cannot pass a unit name through
cctl logsJournals, for one unit or for all of them, with --since, --follow and --unit kernel. Limits: 10000 records, and one hour of following
cctl restart --unit k0sworkerAt corium:operator. You can read units that run once, for example the bootstrap, but you cannot restart them
cctl upgrade <nodes...> --imageOne node at a time. It stops at the first node that does not come back on the digest that it received
cctl rollback <node>Marks the previous image as the next to boot, and does not reboot
cctl cordon / cctl drainAt corium:operator, on controllers only. If a pod disruption budget refuses the drain, the command does not force it, and the node stays cordoned
cctl reboot / cctl shutdownAt corium:admin. The node answers before it goes down
cctl kubeconfig <node>At corium:admin. Points at the virtual IP of the cluster, if there is one. --server overrides it
cctl config <node>At corium:admin. Prints the corium: document that the node recorded when it applied it. cctl apply compares against the same baseline. It does not print the file in /etc, which somebody can edit later. The command tells you when the two disagree
cctl worker-config <controller>At corium:admin, on controllers only. Creates a new worker token and prints the corium: block to paste into a cloud-config. --name, --label and --expiry fill in the block
cctl reset --confirm <node name>In this order: leaves the cluster, erases the bootstrap, forgets its owner, and reboots unclaimed. If a staged image is waiting, the reboot uses it
cctl ca rotate <nodes...> --to <dir>At corium:admin. A node refuses a CA unless the caller shows a certificate signed by it. Thus a rotation cannot lock you out
cctl pki init, cctl pki issueCreate the operator CA and sign client certificates. The CA key stays on your machine
corium-agent api set-ca --fileIf you lose the operator CA key, run this as root on the node. Local only. The node stays a member of its cluster

Why cctl drain is controllers only. A worker holds kubelet credentials, and these cannot evict pods. To drain a worker, use kubectl drain with the kubeconfig that cctl kubeconfig gets.

Where configuration comes from

FeatureNotes
cloud-initNoCloud, ConfigDrive, OpenStack, EC2, Azure, GCE, Hetzner, VMware, OVF
A file on the machine/etc/corium/config.yaml
The kernel command linecorium.config=<url or path>, for PXE
An image default/usr/share/corium/config.yaml

Passthrough

You set everything else that k0s exposes through k0s.patch. Corium applies the patch verbatim to the rendered k0s.yaml after it is done. It merges maps key by key and replaces lists in full.

A setting goes here when few clusters need it, when it is specific to one cluster, or when upstream changes it faster than a schema can follow. A modelled field is a promise to support that spelling forever. That promise is expensive, so Corium makes it only for a small number of settings.

This was tested. The example below renders with every field intact, next to the values that Corium computed:

corium:
  role: controller+worker
  k0s:
    patch:
      spec:
        network:
          kubeProxy:
            mode: ipvs            # or nftables, userspace, or disabled
          nodeLocalLoadBalancing:
            enabled: true         # Envoy on each worker, no external LB
            type: EnvoyProxy
          dualStack:
            enabled: true
            IPv6podCIDR: fd00::/108
        workerProfiles:
          - name: default
            values:
              maxPods: 150
        images:
          repository: registry.example.com/mirror
        featureGates:
          - name: SomeGate
            enabled: true
        konnectivity:
          agentPort: 8132

Commonly reached this way:

Areak0s configurationReference
kube-proxy mode, or disabling itspec.network.kubeProxynetworking
Node-local load balancingspec.network.nodeLocalLoadBalancingNLLB
IPv6 and dual-stackspec.network.dualStackdual-stack
kube-router tuningspec.network.kuberouternetworking
Calico tuningspec.network.caliconetworking
Cluster domainspec.network.clusterDomainconfiguration
API server arguments and portsspec.api.extraArgs, port, k0sApiPortconfiguration
Scheduler and controller-manager argumentsspec.scheduler, spec.controllerManagerconfiguration
etcd tuning and external etcdspec.storage.etcdconfiguration
Kubelet profilesspec.workerProfilesworker profiles
Registry mirroringspec.imagesairgap
Feature gatesspec.featureGatesconfiguration
Konnectivity portsspec.konnectivityconfiguration
Certificate lifetimesspec.api.ca, spec.storage.etcd.caconfiguration
Multiple controllers on kinespec.storage.kine.dataSource pointing at MySQL or PostgreSQLconfiguration

What a patch can and cannot do

Avoid spec.extensions.storage. From k0s 1.31, it is deprecated and does nothing. It still parses, so a configuration that uses it looks accepted but has no effect. Install storage as an add-on instead.

A patch can override anything Corium computed, including values that Corium needs to work. This makes the patch an escape hatch and not only a suggestion. Corium checks only that the result is valid YAML.

Your document stays an ordinary cloud-config. write_files, runcmd, users and every other module continue to work. Corium is a guest in that document, not its owner.


Static configuration, not dynamic

k0s can run in two modes. By default, it reads k0s.yaml from disk on every controller. With --enable-dynamic-config, it stores a ClusterConfig resource in the cluster instead, and controllers reconcile against that resource.

Corium writes a static configuration file. It does not pass --enable-dynamic-config. This has two results:

  • To change the configuration, you reprovision the node. This agrees with the rest of the design: you rebuild a node from an image, you do not adjust it in place.
  • k0s documents a conflict when some controllers use dynamic configuration and others do not. If you enable it, enable it on all controllers.

If you use dynamic configuration, three parts of the configuration always stay in a local file on each controller: spec.api, spec.storage, and spec.network.controlPlaneLoadBalancing. Thus they must be identical on all controllers. Corium already makes sure of this, because every controller in an HA cluster renders the same file.

See k0s: dynamic configuration.


Out of scope

These are decisions, not gaps. Each one is easy to add, but Corium must not add any of them.

Not supportedWhy
k0s AutopilotAutopilot upgrades Kubernetes by replacing the k0s binary on disk. The Corium binary is in the read-only /usr, because Kubernetes ships with the OS. Two upgrade mechanisms that can each change the version independently are worse than one
Fleet managementCorium provisions a node and stops. It does not track, group, or reconcile machines. That is the job of a control plane. See the note below
A configuration API for machines in serviceEvery cloud, hypervisor, and PXE setup already supports cloud-init. That is its value, and Corium does not replace it with a protocol of its own. cctl apply writes /etc/corium/config.yaml. This file is first in the source chain, and it exists for the machines that cloud-init never reaches. Thus cctl apply fills a slot that the design already reserved. It does not add a new authority. The part that the heading names stays out of scope: a node whose role, cluster, name, network, or disks can change under a running Kubernetes. A bootstrapped node re-applies only the safe subset, and refuses each of those fields by name. Nothing watches a document between one apply and the next. cctl config reads the record of a node but cannot change it. See ADR 4 and ADR 8
Arbitrary command execution over the APIThere is no exec endpoint and no endpoint that accepts a command line. Such an endpoint makes the API an SSH with a worse client. Then no argument for a small API surface applies
Forking or patching k0sCorium configures upstream k0s. With a fork, Corium owns the Kubernetes bugs, and that work has no value for the project
Removing add-onsThe k0s Helm extensions install charts. Corium does not model the removal of a chart. If you remove a chart from the configuration, the release continues to run. To remove it, run kubectl delete chart <name> -n kube-system, as k0s: Helm charts describes. For the same reason, cctl apply refuses a day-two document that removes a chart. It does not report a removal that did not happen
Multiple Kubernetes distributionsOnly k0s. Support for k3s or RKE2 too needs an abstraction that fits none of them correctly

On fleet management. The management API does not change that answer. It is one daemon per node that answers for that node. It has no registry, no inventory, and no desired state. A rolling upgrade is a cctl loop over the addresses that you supply.

On removing add-ons. If you delete a chart from the configuration, the release stays in place. Thus cctl apply refuses a day-two document that removes a chart. It does not report a removal that did not happen.

Not supported yet

These features are absent because nobody needed them enough, not because they are wrong.

MissingNotes
Air-gapped bundlesk0s supports air-gap installs with an image bundle in <data-dir>/images/. You can already bake a bundle into the Corium image with a COPY. But there is no corium: field for it, and it is not tested
Worker profiles as a modelled fieldReachable through the patch

Defaults

Where k0s has a default, Corium uses the same default. Thus a Corium node and a k0s node configured by hand agree, unless there is a reason to differ.

SettingCoriumk0s
podCIDR10.244.0.0/16same
serviceCIDR10.96.0.0/12same
CNI providerkuberoutersame
Storage, multi-nodeetcdsame
Storage, role: singlesqlite (kine)implied by --single
Telemetryoffoff
Cluster namecoriumk0s
Add-on namespacedefaultsame

The only intentional difference is the cluster name. It affects only the generated kubeconfig contexts.

Some values that k0s sets keep their k0s defaults, because Corium does not change them. These are the API port 6443, the join API port 9443, the konnectivity ports 8132/8133, the cluster domain cluster.local, and the certificate lifetimes. To see the full set for your version, run k0s config create on a node.