HA cluster

This setup has three controllers, and each one is a full member. The three controllers run etcd together and share a virtual IP. A virtual IP is an address that moves to another machine when its current machine stops. Clients connect to the virtual IP and do not know which machine answered.

This page builds the cluster by hand, because the order of the steps is more important than the commands. A script at the end of the page does all the steps on Proxmox. Read this page first, and the script is easier to understand.

Before you start

Three machines on one broadcast domain. Each machine boots from the Corium image, with one of the configurations below as its cloud-init user data. They can be three VMs, three cloud instances, or three physical machines. Proxmox and OpenStack each show how to create one. The script at the end of this page creates all three.
Four addresses on that segment: one for each machine, and one for the virtual IP. The four decisions in the next section are about these addresses.
The Corium qcow2 or ISO for your platform. See downloads.
SSH access to all three machines from the host where you run step 4, as the user in the users: block.
cctl, for the last section only. Downloads shows how to install it.

The shape


Three controllers	`role: controller+worker`, or `controller` to keep workloads off them
etcd	`storage.type: etcd`. This is the default for controllers, and it is required for more than one controller
A virtual IP	The controller that wins the VRRP election holds it. VRRP is the protocol that picks one holder for the address
`cluster.endpoint`	The virtual IP, on every node

Three is the smallest number of controllers that can lose one and continue. etcd needs a majority of its members. With two members, there is no majority when one of them stops.

Decide four things first

An address for each controller, and a fourth for the virtual IP. All the addresses must be on the same broadcast domain. The controllers use VRRP to agree which one holds the virtual IP, and VRRP traffic does not go through a router.

A virtual router ID, 1–255, unique on that segment. If two clusters use the same ID, each one takes the addresses of the other. If you do not set it, k0s starts at 51. That works until a second cluster on the segment uses the same ID.

A VRRP password of at most eight characters. keepalived cuts a longer password and gives no warning. Two controllers can then believe that they share a password when they do not. For this reason, Corium rejects a longer value.

Whether your network carries multicast. Most clouds do not. If your network has no multicast, list the other controllers under unicastPeers. On a flat LAN, you can set unicastPeers too. It does no harm.

The problem this has to solve

A k0s cluster has no pre-shared secret. The first controller creates the cluster certificate authority (CA). You can create a join token only after the CA exists. For this reason, you cannot give a token to the second and third controllers in advance. The token does not exist when they boot.

Corium makes them wait. If you configure them with join.tokenFrom.waitFor, they boot, find no token, and wait until a token appears. All three machines can start at the same time. The token must arrive before the timeout.

The machines share nothing else. No configuration file holds certificates, a CA, or key material. k0s sends the CA to a joining controller through its join API on port 9443, after the token proves that the controller can join.

1. The first controller

#cloud-config
corium:
  role: controller+worker
  cluster:
    name: homelab
    endpoint: 192.168.0.200        # the virtual IP, not this node's address
  ha:
    enabled: true
    virtualIP: 192.168.0.200/24    # with a prefix: keepalived needs it
    virtualRouterID: 51
    authPass: s3cr3t42             # eight characters or fewer
    unicastPeers:
      - 192.168.0.202
      - 192.168.0.203
  storage:
    type: etcd

Add the users: block from Proxmox step 3. It is the same block. Without it, the machine boots with no account for SSH. examples/ha-controller-first.yaml holds the whole document in one file, with comments.

endpoint is the virtual IP because the value goes into the certificate of the API server. If you set it to the address of one controller, the certificate stops matching as soon as the API moves to another address.

2. The other two

Use the same file. Add a join: block, and set unicastPeers to the other two controllers:

  join:
    tokenFrom:
      file: /etc/corium/join-token
      waitFor: 20m

waitFor lets them boot before the token exists. Without waitFor, a missing token is a hard failure and the node stops. examples/ha-controller-join.yaml shows the whole document.

3. Create them and start all three

Create each machine with its configuration as user data. Then start it, for example with qm start or openstack server create, or turn on the hardware. The order does not matter. The first controller creates the cluster, and the other two wait.

4. Mint a token and deliver it

Warning. Give the token a short expiry, and use a secret store instead of instance metadata when you can. A controller token that has not expired is a cluster-admin credential. A person who holds one can join a full control-plane member with read and write access to etcd. tokenFrom also accepts a url, and a url is the better choice in production.

When the first controller answers, create the token and store it:

ssh core@192.168.0.201 sudo k0s status          # wait for this to succeed
TOKEN=$(ssh core@192.168.0.201 sudo k0s token create --role=controller --expiry=1h)

Then write it to /etc/corium/join-token on each of the other two controllers, with mode 0600:

for ip in 192.168.0.202 192.168.0.203; do
  ssh "core@${ip}" "sudo install -m600 /dev/stdin /etc/corium/join-token" <<< "$TOKEN"
done

Before you send the token, make sure that $TOKEN is not empty. A waiting controller does not treat an empty file as an error. It waits for the full waitFor time and gives no message that shows the cause.

Each waiting controller finds the file and joins. Then k0s sends it the CA.

5. Check it

$ sudo k0s kubectl get nodes -o wide
NAME              STATUS   ROLES           VERSION       INTERNAL-IP
corium-7c17d087   Ready    control-plane   v1.36.4+k0s   192.168.0.201
corium-87025d9c   Ready    control-plane   v1.36.4+k0s   192.168.0.202
corium-5c5d785f   Ready    control-plane   v1.36.4+k0s   192.168.0.203

$ sudo k0s etcd member-list
{"members":{"corium-5c5d785f":"https://192.168.0.203:2380", ...}}

Each node registers its own address, not the virtual IP. This is necessary. A cluster where the nodes register the virtual IP works only until the first failover.

The API answers on the virtual IP. Point your kubeconfig at this address:

$ curl -sk -o /dev/null -w '%{http_code}\n' https://192.168.0.200:6443/readyz
401

401 is the correct answer. The server is running and refuses a request that has no authentication.

6. Watch it fail over

The address does not show which controller holds the virtual IP. ARP shows it:

ip neigh flush dev <your bridge>
ping -c2 192.168.0.200 >/dev/null
ip neigh show 192.168.0.200          # compare the MAC against each node's

Stop that controller and run the commands again. In a real test, the virtual IP moved in less than five seconds. The API continued to answer, etcd kept its majority (quorum) with two of three members, and the cluster continued to schedule. When the controller started again, it was Ready in less than fifteen seconds.

Doing all of that on Proxmox

deploy/proxmox/create-ha-cluster.sh does every step above. It writes the three configurations and creates a VRRP password of exactly eight characters. It creates and starts the VMs, waits for the first controller, creates a token, and sends the token to the other two. Copy it to the host the same way that Proxmox does, and run it from deploy/proxmox/.

Warning. Read these two points before you run the script. First, SSH_KEY must be a key whose private half is on this Proxmox host. The script uses SSH from this host to send the token. If you use a key from your laptop, the script stops at Permission denied after it built the three VMs. Second, the script destroys any VM that already uses one of those VMIDs. create-vm.sh does not do this.

DISK_IMAGE=/var/lib/vz/template/corium-0.5.0-x86_64.qcow2 \
SSH_KEY="$(cat ~/.ssh/id_ed25519.pub)" \
CLUSTER_NAME=homelab VIP=192.168.0.200 GATEWAY=192.168.0.1 \
NODE_IPS="192.168.0.201 192.168.0.202 192.168.0.203" \
VMIDS="142 143 144" \
  ./create-ha-cluster.sh

When you are done

for id in 142 143 144; do qm stop "$id"; qm destroy "$id" --purge; done
rm -f /var/lib/vz/snippets/corium-14*.yaml
for ip in 192.168.0.201 192.168.0.202 192.168.0.203; do
  ssh-keygen -f /root/.ssh/known_hosts -R "$ip"
done

qm destroy does not remove the snippets, and the host keeps the old host keys of the controllers. If you do not remove the keys and you rebuild on the same addresses, SSH shows REMOTE HOST IDENTIFICATION HAS CHANGED.

Managing it afterwards

Two tasks on an HA cluster are easier with the management API than by hand. Both are about the virtual IP.

A kubeconfig that points at one controller stops working when that controller stops. The cluster continues to run, but your access to it does not. cctl kubeconfig <any controller> points the file at the virtual IP, because the node recorded the virtual IP when it bootstrapped:

cctl kubeconfig 192.168.0.201 > kubeconfig
grep server: kubeconfig          # https://192.168.0.200:6443

An upgrade of one controller at a time is most important with three controllers, because two of them make the quorum:

cctl upgrade 192.168.0.201 192.168.0.202 192.168.0.203 --image ghcr.io/corium-os/corium:<tag>

If a controller does not come back, the upgrade stops there. It does not take down the second controller.

Both tasks need api.operatorCA in the configuration of each controller. The API runs without it, and ADR 11 made that the default. But a controller that names its operator CA is claimed at boot. Without it, the controller is open to enrolment for the minute that the bootstrap takes. The operator CA is what lets a three-node build run with no operator present.

Docs

Corium

Title here

HA cluster

Before you start

The shape

Decide four things first

The problem this has to solve

1. The first controller

2. The other two

3. Create them and start all three

4. Mint a token and deliver it

5. Check it

6. Watch it fail over

Doing all of that on Proxmox

When you are done

Managing it afterwards

What to read next

HA cluster

Before you start#

The shape#

Decide four things first#

The problem this has to solve#

1. The first controller#

2. The other two#

3. Create them and start all three#

4. Mint a token and deliver it#

5. Check it#

6. Watch it fail over#

Doing all of that on Proxmox#

When you are done#

Managing it afterwards#

What to read next#

Before you start

The shape

Decide four things first

The problem this has to solve

1. The first controller

2. The other two

3. Create them and start all three

4. Mint a token and deliver it

5. Check it

6. Watch it fail over

Doing all of that on Proxmox

When you are done

Managing it afterwards

What to read next