How to Setup K3s Kubernetes with IPVS

IPVS (IP Virtual Server) provides efficient load balancing compared to iptables. A K3s setup is created to have support for IPVS. This guide teaches you how to install K3s with IPVS and correctly configure it.

Prerequisites

To follow along with this K3s and IPVS guide ensure you have the following:

• A set of nodes (VMs or physical machines) for your K3s cluster.
• Root or sudo access on all nodes.
• Basic understanding of Kubernetes and container orchestration.

K3s and IPVS

K3s is optimized to let your environments use limited resources. Any Kubernetes resource is compatible with K3s. You just need minor twerks to get up and running. IPVS will add:

• Act as a transport layer for load balancing into the Linux kernel.
• It acts as a load-balancing IP layer, to handle network traffic.
• You get good balancing algorithms like round-robin, least connection, and destination hashing.

Configuring IPVS

Once you have updated your dependencies and the curl is ready, you will only add a funnel network to K3s. First, get the dependencies ready:

sudo apt update
sudo apt install -y curl

The first step is to Install K3s on the Master Node. However, you will need to verify kernel modules are loaded:

lsmod | grep br_netfilter
lsmod | grep overlay

If you get no output, use these commands to load the kernel modules manually:

sudo modprobe br_netfilter
sudo modprobe overlay

Rerun the command and check the results:

lsmod | grep br_netfilter
lsmod | grep overlay

Output:

br_netfilter           32768  0
bridge                421888  1 br_netfilter
overlay               212992  0

Go further and check your kernel has IPVS support:

lsmod | grep ip_vs

If again you get no results load the IPVS modules with these commands:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh

Your expected lsmod | grep ip_vs results will be as such:

$ lsmod | grep ip_vs
ip_vs_sh               12288  0
ip_vs_wrr              12288  0
ip_vs_rr               12288  0
ip_vs                 225280  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          200704  6 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
libcrc32c              12288  4 nf_conntrack,nf_nat,nf_tables,ip_vs

Installing K3s with IPVS

Here you will download and install K3s with the IPVS backend as the flannel network.. Use this command:

• Ensure you Uninstall K3s, you have a previos Node:

sudo /usr/local/bin/k3s-uninstall.sh

• Use these commands to get K3s ready with ip_vs:

curl -sfL https://get.k3s.io | sh -s - server --kube-proxy-arg proxy-mode=ipvs 

This should install K3s and enable IPVs all at once:

If you want to set the K3s on Worker Nodes use:

curl -sfL https://get.k3s.io | K3S_URL=https://<master-node-ip>:6443 K3S_TOKEN=<node-token> sh -

Replace <master-node-ip> with the IP address of your master node and <node-token> with the token from your master node (/var/lib/rancher/k3s/server/node-token).

If K3s is ready and running with IPVS, verify the kube-proxy configuration:

kubectl get nodes

## Output
    mode: "ipvs"

You may get the following error:

WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start the server with --write-kubeconfig-mode or --write-kubeconfig-group to modify kube config permissions 
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

Reinstall K3s with this command:

curl -sfL https://get.k3s.io | sh -s - server --kube-proxy-arg proxy-mode=ipvs --write-kubeconfig-mode 644

I have created this Solving unable to read etc rancher k3s k3s yaml K3S Error in case you need more details.