Corium Get started

Cilium

This page replaces the CNI of a node with Cilium. It shows what to put in the cloud-config, how to make sure that the install worked, and what changes if you also remove kube-proxy. Corium ships kube-router because k0s ships it. That one component does networking, network policy and service proxying. To replace it, you make two decisions.

This page applies to a node that you are about to boot, not to a cluster that already runs. Corium renders cni: custom into /etc/k0s/k0s.yaml at bootstrap. To change it later, you must edit that file and restart k0s.

For a single node, start at Configure the node. For multiple nodes, read Multi-node clusters first. The CNI is a decision for the whole cluster, and you must declare the chart on the controller only.

Both decisions go in the cloud-config that the node already boots with. First, k0s must install no CNI, so that two CNIs do not conflict on the same node. Second, something must install Cilium. cni: custom does the first, and an entry under addons: does the second. The node then gets to the correct state without help, and you do not need a second provisioning step.

Configure the node

#cloud-config
corium:
  role: single

  network:
    podCIDR: 10.244.0.0/16
    cni: custom

  addons:
    - name: cilium
      chart: cilium/cilium
      version: 1.20.2
      namespace: kube-system
      repository:
        name: cilium
        url: https://helm.cilium.io/
      values:
        ipam:
          mode: cluster-pool
          operator:
            clusterPoolIPv4PodCIDRList:
              - 10.244.0.0/16

This is a complete node. examples/custom-cni.yaml is the same document with an SSH key, ready to boot.

Before you boot anything, validate the document:

corium-agent validate node.yaml
node.yaml: valid (role single, cluster corium)

What cni: custom and addons: do

cni: custom renders spec.network.provider: custom into /etc/k0s/k0s.yaml. k0s then deploys no CNI. The node registers, comes up NotReady, and stays NotReady until Cilium runs. This state is expected, and it is not a failed boot. See troubleshooting to tell the two apart.

addons: renders spec.extensions.helm. k0s turns this into Chart resources that its own controller reconciles, with no Helm binary and no in-cluster operator. The controller runs the install, not a pod. So the install works while the cluster has no pod network yet, and this is why you can install a CNI this way. The Cilium agent and operator both run on the host network, so they do not need the network that they will provide.

corium-agent bootstrap --dry-run shows what the two fields produce, and it does not change the machine. The output below shows only the parts that this page is about:

# /etc/k0s/k0s.yaml
spec:
    extensions:
        helm:
            charts:
                - chartname: cilium/cilium
                  name: cilium
                  namespace: kube-system
                  values: |
                    ipam:
                        mode: cluster-pool
                        operator:
                            clusterPoolIPv4PodCIDRList:
                                - 10.244.0.0/16
                  version: 1.20.2
            repositories:
                - name: cilium
                  url: https://helm.cilium.io/
    network:
        podCIDR: 10.244.0.0/16
        provider: custom
        serviceCIDR: 10.96.0.0/12

Corium passes the chart values through without changes. It does not compare them with the rest of the configuration. You must make sure that clusterPoolIPv4PodCIDRList agrees with network.podCIDR. If they disagree, pods get addresses that the cluster does not route.

Nothing to configure for the immutable filesystem

On a read-only OS, two items usually need work. Corium already handles both:

  • /opt/cni/bin is writable. The image points /opt at /var/opt, because the CNI DaemonSet installs its binary there. Without this, the first error you see is mkdir /opt/cni: read-only file system. The chart defaults, /opt/cni/bin and /etc/cni/net.d, are already where k0s looks. You do not need to set cni.binPath and cni.confPath.
  • The inotify limits are higher. On a stock host, cilium connectivity test fails with too many open files. The image ships the higher values in /usr/lib/sysctl.d/.

Verify the install

k0s kubectl get charts -A                                 # did the chart install
k0s kubectl -n kube-system get pods -l k8s-app=cilium     # is the agent running
k0s kubectl get nodes                                     # Ready, once it is

A few minutes after boot, all three of these must be true:

  • get charts -A lists k0s-addon-chart-cilium in kube-system. If it is missing, k0s did not see the add-on. Make sure that the addons: block is correct.
  • The Cilium agent pod is Running and 1/1 ready, one per node. Pending means that nothing scheduled it. CrashLoopBackOff means that it started and failed, so read its logs.
  • The node is Ready, approximately one minute after the agent reports healthy. Until then, NotReady is expected, because nothing installed a network yet.

If you install the Cilium CLI, cilium status --wait gives more information.

Without kube-proxy

Cilium can do all of the service handling. This needs two changes, and both must be in the configuration that the node first boots with. To remove kube-proxy from a cluster that already runs it is a different task, and a more difficult one.

corium:
  role: controller+worker

  cluster:
    endpoint: 10.0.0.10

  network:
    cni: custom

  # kube-proxy has no corium: key. k0s.patch is applied verbatim to the
  # rendered k0s.yaml, so every k0s setting stays reachable.
  k0s:
    patch:
      spec:
        network:
          kubeProxy:
            disabled: true

  addons:
    - name: cilium
      chart: cilium/cilium
      version: 1.20.2
      namespace: kube-system
      repository:
        name: cilium
        url: https://helm.cilium.io/
      values:
        kubeProxyReplacement: true
        k8sServiceHost: 10.0.0.10
        k8sServicePort: 6443

Here, k8sServiceHost is required. Cilium must reach the API server to load its own configuration. Usually it uses the in-cluster kubernetes Service, but that Service is a ClusterIP, and nothing translates it yet. kube-proxy is gone, and Cilium is its replacement. Give Cilium an address that works without a cluster network: the address of a controller, or the HA virtual IP.

Multi-node clusters

The CNI is a decision for the whole cluster, and the control plane makes it:

  • Declare the chart on the controller only. If a worker declares addons:, validation rejects it with addons: only a controller installs add-ons, but role is "worker". The worker does not ignore it without a message.
  • Put network: on the controller too. Corium writes the k0s cluster configuration on controllers only, so it never renders the cni: value of a worker.
  • Workers need nothing. A worker that joins later is NotReady until the DaemonSet starts on it, then Ready. There is no step for each node.

Troubleshooting

The node is NotReady and there are no Cilium pods. The chart did not install. k0s kubectl get charts -A shows what k0s thinks it did, and journalctl -u k0scontroller | grep -i helm tells you why it did not. The usual cause is a version that does not exist. The second cause is a node with no route to helm.cilium.io, because k0s fetches the repository at install time.

Cilium crash-loops with Unable to contact k8s api-server. With kubeProxyReplacement: true, the cause is a missing or wrong k8sServiceHost. The agent tries to reach the API server through a Service IP that nothing translates.

Pods get addresses that the cluster does not route. clusterPoolIPv4PodCIDRList and network.podCIDR disagree. Nothing makes sure that they agree, so you must do it.

Everything works, but you expected kube-router. cni: custom stays after bootstrap. Corium renders it into /etc/k0s/k0s.yaml at bootstrap. To change it later, edit that file and restart k0s. A reboot does not change it.

Other CNIs

Only the values on this page are specific to Cilium. You install any CNI that comes as a chart with cni: custom and an add-on. Look for the same two failures: a chart that does not install, and a pod CIDR that disagrees. Calico is the exception. It has its own value, cni: calico, and k0s installs it for you.

The configuration reference describes every field on this page. Feature support lists what else k0s.patch can reach.

Prev
Upgrades
Next
OIDC login