Part 6 - Performance Tuning: Getting the Most from Raspberry Pi

---

Why Tune?

The truth is, my 11-node Raspberry Pi 4B K3s cluster is not a rack of Xeon-powered space heaters. Each Pi is a 4-core ARM Cortex-A72 with 2-8GB RAM. In this setup, there are a lot of things that matter such as thermal limits, power stability, SD card wear, USB storage nuances, network bottlenecks, ARM image compatibility, and the overhead of K3s itself. The default configuration we have currently leaves performance on the table and wastes power on features that my cluster does not use.

Once I got the stack working, I started some tuning to make it run efficiently for the long term. here is what I finally ended up with

---

Orderly Shutdown Before Changes

Any time firmware or kernel configuration changes need to be applied to the cluster, ensure that the stack is restarted (reboot or shutdown/startup) gracefully. This is important for the stability of our cluster, its nodes, and the pods that run on them.

I started by cordoning all the nodes, followed by draining them, in an order (FYI - cordining a node makes it SchedulingDisabled. Draining gracefully evicts the pods running on a node):

# Cordon all nodes
kubectl cordon $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}')

# Drain worker nodes first
for node in $(kubectl get nodes --selector='!node-role.kubernetes.io/control-plane' \
  -o jsonpath='{.items[*].metadata.name}'); do
  kubectl drain $node --ignore-daemonsets --delete-emptydir-data --timeout=60s
done

--ignore-daemonsets flag is required here because my cluster has DaemonSet-managed pods (the MetalLB speaker pods in the metallb-system namespace). By default, kubectl drain will fail if DaemonSet-managed pods are present, because those pods cannot be evicted like normal workload pods. The flag tells drain to ignore them and continue draining.

# Now, Drain the head node
kubectl drain headnode --ignore-daemonsets --delete-emptydir-data --timeout=60s

Remember - Worker nodes first, Head node last, ALWAYS!

The head node serves NFS and if it goes down while workers are still running, they lose their storage mounts and the results may not be pretty. Also, if your head node is the one and only K3s server in your cluster (like how I do), treat it as the final shutdown target rather than just another node.

On startup, do the reverse: get the head node up and running first, wait for NFS to be ready, SSH in and verify, then start bringing back the worker nodes.

It may be a good idea to run some basic checks before bringing all the worker nodes back online:

# Validate nodes
kubectl get nodes -o wide

# Validate pods
kubectl get pods -A -o wide

# Validate PVCs
kubectl get pvc -A

# Show the 50 most recent Kubernetes events across all namespaces
kubectl get events -A --sort-by='.lastTimestamp' | tail -50

# Verify NFS status and exports on the head node
sudo systemctl status nfs-server
sudo exportfs -v

Once all nodes are up and running, uncordon them:

kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}')

---

CPU Governor

By default, Raspberry Pi OS uses ondemand as the CPU scaling governor. It's conservative, ramps up slowly and doesn't always react fast enough to short DNS query bursts.

schedutil is better for this workload. It uses CPU utilization signals from the kernel scheduler to make faster, more accurate scaling decisions. In my cluster, the difference was subtle but real - query latency spikes were less common under sudden load.

Set it persistently via udev, which fires on every boot (create a custom udev rule):

# /etc/udev/rules.d/60-cpu-governor.rules
SUBSYSTEM=="cpu", ACTION=="add", ATTR{cpufreq/scaling_governor}="schedutil"

This overrides the distribution default without modifying any system files that might get overwritten during upgrades.

Now, reload the udev rules from disk:

sudo udevadm control --reload-rules

Then tell udev, the Linux device manager, to trigger device events for devices in the cpu subsystem:

sudo udevadm trigger --subsystem-match=cpu

Now verify the setting:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# schedutil
# schedutil
# schedutil
# schedutil

If the setting still has not changed, reboot the node after following the cordon and drain steps discussed earlier. Once the node is back up and the CPU governor setting is verified, remember to uncordon the node.

---

Overclock Configuration

The RPi 4B can be overclocked reliably (as long as you have adequate cooling and stable power) using the over_voltage_delta mechanism introduced in a recent firmware release. Unlike the legacy over_voltage setting, over_voltage_delta specifies voltage in microvolts relative to the stock value, which makes the adjustment more explicit and precise.

Apply the following settings to /boot/firmware/config.txt on each node:

4GB and 8GB workers (can handle full 2GHz):

arm_freq=2000
arm_freq_min=600
over_voltage_delta=50000
gpu_mem=16
dtoverlay=disable-wifi
dtoverlay=disable-bt
hdmi_blanking=2

Head node and 2GB worker (conservative):

arm_freq=1800
arm_freq_min=600
over_voltage_delta=25000
gpu_mem=16
dtoverlay=disable-wifi
dtoverlay=disable-bt
hdmi_blanking=2

gpu_mem=16 minimizes memory reserved for the GPU. My cluster is headless, so there is no reason to reserve more GPU memory than necessary.

disable-wifi and disable-bt save power and eliminate RF interference. We don't use these services anyway.

hdmi_blanking=2 powers down the HDMI output completely.

After rebooting, temperatures and frequencies across my cluster were:

Node	Temp	Freq
headnode	33°C	1200 MHz
node1-node4	30-36°C	1500-1700 MHz
node5-node9	33-42°C	1100-1700 MHz
node10	42°C	1700 MHz

As you can see, all were well below the 80°C to 85°C throttling range. The schedutil governor scales up to 2000MHz under load and backs off at idle.

---

Kernel Parameters

A few sysctl settings can make a meaningful difference for a cluster running NFS storage and DNS workloads:

# /etc/sysctl.d/99-k3s.conf
net.core.rmem_max=2500000
net.core.wmem_max=2500000
net.core.netdev_max_backlog=5000
net.ipv4.tcp_fastopen=3
vm.dirty_ratio=15
vm.dirty_background_ratio=5
vm.swappiness=1
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=512

The network buffer increases help with NFS throughput. vm.swappiness=1 tells the kernel to avoid swapping almost entirely. On nodes where swap is disabled, this is a no-op, but on the head node it discourages swapping under pressure. The inotify limits matter for K3s, which creates many file watches for its embedded components.

---

Disabling Unused Services

Several services that start by default were doing nothing useful on my headless cluster nodes:

sudo systemctl disable --now avahi-daemon bluetooth wpa_supplicant nfs-blkmap

• avahi-daemon: mDNS/Bonjour discovery. Not needed; actually generates DNS traffic that shows up in Pi-hole logs as noise.

• bluetooth: disabled in firmware too, belt-and-suspenders.

• wpa_supplicant: WiFi daemon. WiFi is disabled in firmware. Our cluster is all wired.

• nfs-blkmap: pNFS block layout daemon. Our setup uses regular NFS, not pNFS.

---

Journal Configuration

By default, systemd-journald writes logs to disk on every node. On SD cards, this is unnecessary write wear. Worker nodes do not need long-term persistent journals for normal application debugging. Most workload logs are available through K3s with kubectl logs, and short-lived node logs are still available in the RAM while the node is running.

Workers use volatile (RAM) journals:

# /etc/systemd/journald.conf.d/99-volatile.conf
[Journal]
Storage=volatile
RuntimeMaxUse=64M
RuntimeMaxFileSize=16M
Compress=yes

The head node keeps persistent journals; that is where debugging happens:

# /etc/systemd/journald.conf.d/99-persistent.conf
[Journal]
Storage=persistent
SystemMaxUse=200M
SystemMaxFileSize=50M
Compress=yes

---

Results

After applying all tuning config settings:

• Temperatures: 30-44°C across all nodes under normal load

• Frequencies: scaling between 600MHz (idle) and up to 2000MHz (load) via schedutil

• Did not see any thermal throttling

• DNS query response times stayed consistently under 10 ms

The cluster has been running continuously without intervention for 4 days now. The performance tuning was a one-time investment that I believe will pay for itself in longevity.

---

Next: What I Learned, What I'd Do Differently

---

Cover image generated by AI. The fortress may be fake, but the over-engineering is very real.

All YAML files referenced in this series are available in the companion GitHub repository.