Part 1 - The Problem, The Plan, and The Foundation

Because one Pi-hole was too simple, and apparently so was one Raspberry Pi. A multi-part journey into highly available, encrypted, ad-blocking DNS on a K3s Raspberry Pi cluster.

---

The Problem with Home DNS

Every device on your home network - your phone, your smart TV, those fancy security cameras, thermostats, your refrigerator that apparently needs internet access - makes dozens of DNS queries every minute. DNS is the phone book of the internet: before your browser can load a page, it has to ask a DNS server, “What’s the IP address for google.com?”

Most home networks handle this with whatever DNS server the router provides, which typically forwards to your ISP's resolvers or a known public one like Google or Cloudflare. This works, but it has three problems.

Problem 1: Single point of failure. One DNS server goes down and your entire network effectively loses internet access. It doesn't matter that Google's servers are fine. If the device answering DNS queries in your house stops responding, nothing works. Every browser shows an error. Every app times out. Your spouse is not impressed. Your kids are angry.

Problem 2: Privacy. Standard DNS queries are sent in plaintext. Your ISP can see every domain every device on your network visits. Your router manufacturer can see it. Anyone who can intercept traffic between you and your resolver can see it. This is not a paranoid concern as DNS data is commercially valuable and actively collected.

Problem 3: Ads and trackers. Every ad, tracking pixel, and telemetry ping starts with a DNS query. Ads are annoying. Trackers are worse. They follow you across every site, build a profile of your browsing habits, and sell it to anyone willing to pay. Some ad networks have served malware. Others fingerprint your device. All of them consume bandwidth, slow down page loads, and phone home from every device on your network, including the ones that give you no option to opt out.

There are plenty of ways to solve one of these problems. Browser extensions handle ads on your laptop. A VPN handles some of the privacy concerns. Your router might have a basic firewall. Or you decide that “plug-and-play” is for cowards, and your weekends are better spent staring at firewall logs wondering why your IoT toaster is making friends with a server in Belarus. So you install and configure pfSense.

But solving all three, for every device on the network, without touching any of them individually? That requires controlling DNS at the network level. And once you decide to do that properly with redundancy, encryption, and visibility, the yak shaving begins.

This is the story of how blocking ads on my home network turned into an 11-node Raspberry Pi Kubernetes cluster.

---

What We're Building

By the end of this series, the home network will have:

• Two Pi-hole instances running on K3s, on different physical nodes, behind a single IP address. If one node dies, the other continues serving DNS with little to no visible interruption, and without manual intervention.

• Two DNSCrypt instances that encrypt and authenticate DNS traffic between clients and the configured upstream DNS resolver. This prevents your ISP or other on-path observers from reading or tampering with DNS queries in transit, and helps prevent DNS hijacking and spoofing. However, it does not hide the destination IPs you connect to, so an ISP can still infer some visited sites from IP addresses, reverse DNS, traffic patterns, and sometimes TLS metadata such as SNI. DNSCrypt protects the client-to-resolver leg only; it does not protect the entire browsing session or resolver-to-authoritative DNS traffic.

• Ad and tracker blocking across every device on the network - phones, TVs, IoT cameras, doorbells, laptops - without installing anything on any of them.

• Per-device IP logging showing exactly which device made which DNS query.

• Two physically separated networks. A main network and an IoT network, both protected by the same Pi-hole VIP, with network isolation maintained between them.

• Automatic configuration sync between the two Pi-hole instances so changes propagate without manual work.

Everything runs on K3s, a lightweight, production-ready Kubernetes distribution built for IoT and edge computing, which is exactly our kind of hardware. Not because K3s is required for this (a single Raspberry Pi running Pi-hole would handle 100 devices without breaking a sweat), but because running it on K3s provides workload-level redundancy for Pi-hole and makes the project a much more interesting engineering problem.

---

Kubernetes Naming (and it's Funny)

Before we dive in deep, let's take a quick detour to see why this series uses "K3s" rather than "Kubernetes."

Kubernetes gets abbreviated as k8s because there are exactly eight letters between the “k” and the “s” in “Kubernetes” - a numeronym. The project came out of Google in 2014 (preceded by Borg and Omega), and apparently the engineers decided that typing 10 letters repeatedly was unacceptable.

K3s is a lightweight Kubernetes distribution built for edge computing and resource-constrained hardware like Raspberry Pis. The goal of K3s was to create an installation with roughly half the memory footprint of k8s. As the joke goes, if standard Kubernetes is 10 letters long, then something "half as big" should be 5 letters long. Per the "k-s" format, a 5-letter word should have 3 letters in the middle, hence K3s. There is also a "visual pun" interpretation - an 8 cut vertically in half looks like a 3, hence K3s.

Here is the genuinely funny part: if you cut the "8" in "k8s" horizontally instead of vertically, you get "k0s." This is also a real Kubernetes distribution built by Mirantis.

Those who want to see how the K8s sausage was made can take a trip down memory lane with IBM’s exposé on Kubernetes history.

For our purposes, K3s is the right choice. Single binary, embedded SQLite datastore, runs comfortably on Raspberry Pi 4 hardware, and wastes no resources on cloud-specific components we don’t need.

---

Hardware

The cluster runs on 11 Raspberry Pi 4B nodes housed cozily inside a fancy, custom open-frame rack (two $30 acrylic cluster cases).

Node	RAM	#	Role
headnode	8 GB	1	K3s control plane + NFS server
node1 – node4	8 GB	4	Worker (capacity: max)
node5 – node9	4 GB	5	Worker (capacity: med)
node10	2 GB	1	Worker (capacity: min)

Each node has an official Raspberry Pi PoE+ HAT attached.

Storage: The head node has a 2TB USB 3 SSD. It serves two purposes: storing K3s’s embedded SQLite datastore and acting as an NFS server for the entire cluster. All persistent application data lives on this SSD, mounted into pods over NFS from any node in the cluster. SD cards are far too slow and unreliable for sustained database writes.

Worker nodes boot from SD cards. The OS footprint is small, mostly read-only, and generates minimal write activity, so SD cards are fine for boot media. Anything that needs to survive a reboot goes to the NFS share.

Cooling: An open-frame rack, heatsinks on every Pi, a fan on each PoE+ card and each rack slot, plus two 140 mm fans underneath pushing air across all the nodes. Nodes idle at 30–35°C and peak around 45°C under sustained load, well below the 80°C throttle threshold.

Networking: All nodes connect via Gigabit Ethernet to two "managed" PoE+ switches. WiFi is disabled at the firmware level on every node (it's unused, wastes power, and generates RF noise).

Power: All nodes are powered via the two PoE+ switches. The alternative is finding eleven free outlets, untangling eleven power bricks, and explaining to everyone in the house why the power strip situation has "gotten a little out of hand." PoE carries power and data over a single Ethernet cable per Pi. Highly recommended. Your sanity will thank you, and your rack will have far less cable spaghetti.

---

Operating System

All nodes run Debian GNU/Linux 13 (trixie) arm64, Linux 6.12.75+rpt-rpi-v8 kernel. Fresh install on every node with no desktop environment, no GUI packages, nothing that is not needed for a headless server.

Before touching K3s, each node needs to be properly prepared. These steps apply to every node, head and workers alike. Skipping any of them can and will cause problems later.

Step 1 - Set a static IP

Non-negotiable for cluster nodes. If a node's IP changes, K3s loses track of it. Either configure a static IP directly on the node or create a DHCP reservation in your router. The router reservation approach is cleaner; node configuration stays simple, and IP management stays centralized. Since I use a Firewalla Gold+, it was just a couple of clicks within the iOS app.

If you want cluster nodes to have predictable sequential IPs, assign static IPs outside your DHCP pool but inside the same subnet.

Step 2 - Disable swap

Disable swap to avoid unpredictable latency and keep K3s resource behavior simple. To disable swap on Raspberry Pi OS Trixie, add a drop-in config:

sudo mkdir -p /etc/rpi/swap.conf.d/
sudo tee /etc/rpi/swap.conf.d/99-disable-swap.conf <<EOF
[Main]
Mechanism=none
EOF

Reboot and verify nothing is swapping:

swapon --show
# Should return nothing

Step 3 - Enable cgroups

Kubernetes uses Linux cgroups to manage container resources such as CPU limits, memory limits, and so on. On Raspberry Pi, cgroups must be explicitly enabled in the kernel command line.

Edit /boot/firmware/cmdline.txt. This file contains a single line. Add the following to the end of that line. DO NOT ADD A NEW LINE!:

cgroup_enable=cpuset cgroup_enable=memory

The result should look something like this:

console=serial0,115200 console=tty1 root=PARTUUID=xxxxxxxx rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory

On current Raspberry Pi OS, cgroup_enable=memory is the important part. Many K3s guides also include cgroup_memory=1 for compatibility with older Raspberry Pi kernels. You may need it; I didn’t.

Reboot after this change. Without it, K3s starts but containers won't have proper resource limits and some workloads will behave unexpectedly.

Step 4 - Set hostnames

Each node needs a unique hostname:

sudo hostnamectl set-hostname your-node-name

Or, do this while prepping OS SD cards for each node.

Step 5 - Update the bootloader EEPROM

sudo rpi-eeprom-update
sudo rpi-eeprom-update -a
sudo reboot

The latest bootloader includes better power management, USB stability improvements (critical for the SSD on the head node), and improved thermal control. Worth doing before building anything on top.

Step 6 - Set up passwordless SSH from head node

Configure SSH key-based access from the head node to each worker so cluster management scripts can run non-interactively (otherwise be prepared to enter 11 different passwords a gazillion times). Generate a key on the head node and copy it to each worker:

ssh-keygen -t ed25519 -C "cluster-admin" -f ~/.ssh/id_ed25519 -N ""

for ip in xxx.xxx.xxx.202 xxx.xxx.xxx.203 xxx.xxx.xxx.204 \
          xxx.xxx.xxx.205 xxx.xxx.xxx.206 xxx.xxx.xxx.207 \
          xxx.xxx.xxx.208 xxx.xxx.xxx.209 xxx.xxx.xxx.210 \
          xxx.xxx.xxx.211; do
  ssh-copy-id pi@$ip
done

You'll be prompted for the password once per node. After this, SSH from the head node to any worker requires no password, which is essential for the cluster management scripts we'll use later.

Step 7 - For your sanity

Reboot all nodes.

---

Installing K3s

With prerequisites done on every node, K3s installation is straightforward.

Head node

curl -sfL https://get.k3s.io | sh -s - \
  --write-kubeconfig-mode 644 \
  --disable local-storage \
  --node-name headnode

Three flags worth noting:

--write-kubeconfig-mode 644 - makes the kubeconfig file readable without sudo, so you can run kubectl commands as the pi user without prefixing everything with sudo.

--disable local-storage - disables K3s's built-in local path storage provisioner. Local storage writes directly to the node's SD card which is fine for testing, but a fast path to SD card failure in production. All persistent storage goes to the NFS-backed SSD on the head node instead. You don't want to kill the SD card just yet.

I designed the cluster around NFS-backed persistent volumes. Since workloads can be rescheduled onto different worker nodes, their persistent data needs to be available from any node in the cluster. Local storage is node-bound, so it does not work for workloads that need to survive rescheduling.

--node-name headnode - sets the node name explicitly rather than using the hostname. Keeps cluster node names consistent and predictable. You can also set this in /etc/rancher/k3s/config.yaml later (and restart the k3s service).

After installation, retrieve the join token for workers:

sudo cat /var/lib/rancher/k3s/server/node-token

Worker nodes

Run on each worker, substituting your head node's IP and the token from above (this may take a while, so grab a coffee beforehand):

curl -sfL https://get.k3s.io | \
  K3S_URL=https://xxx.xxx.xxx.201:6443 \
  K3S_TOKEN=YOUR_TOKEN_HERE \
  sh -s - agent \
  --snapshotter=native \
  --node-name node_name

--snapshotter=native - uses containerd’s native snapshotter instead of overlayfs. I chose this because it was the conservative compatibility choice on this Raspberry Pi SD card setup, even though it may be slower and use more disk space than overlayfs. My first attempt used overlayfs, and I occasionally saw containerd/image-pull errors on my Pis, including:

failed to create containerd task: failed to create shim: OCI runtime create failed

and:

unexpected EOF
failed to pull and unpack image "docker.io/pihole/pihole:latest":
  failed to unpack image on snapshotter overlayfs:
    failed to extract layer:
      exit status 1

Full disclosure: these errors were intermittent. Plenty of RPi K3s setups run overlayfs without issues. If you feel adventurous, install K3s agents without this flag and see what happens. If you start seeing the same kind of containerd/image-pull errors I saw, add --snapshotter=native to the agent config and restart k3s-agent.

To me, this was a defensive choice and I keep it in the install command for consistency and to avoid future debugging sessions where the error messages are technically accurate, completely unhelpful, and somehow still your fault.

--node-name node_name - sets the node name explicitly rather than using the hostname. Keeps cluster node names consistent and predictable. You can also set this in /etc/rancher/k3s/config.yaml later (and restart k3s-agent).

Verify

Back on the head node:

kubectl get nodes

All nodes should appear as Ready within a minute or two of joining. If a node stays NotReady, the most common cause is missing cgroup settings in /boot/firmware/cmdline.txt.

---

Node Labels

With 11 nodes of varying RAM, labels control which workloads land where. The key label is capacity:

# Head node - 8 GB, control plane + NFS
kubectl label node headnode capacity=max

# 8 GB workers
kubectl label node node1 node2 node3 node4 capacity=max

# 4 GB workers
kubectl label node node5 node6 node7 node8 node9 capacity=med

# 2 GB worker
kubectl label node node10 capacity=min

Three tiers give fine-grained control over pod placement:

• capacity: max - 8 GB nodes, suitable for memory-hungry workloads

• capacity: med - 4 GB nodes, general purpose workers

• capacity: min - 2 GB node, lightweight or short-lived workloads only

Throughout this series, Pi-hole and DNSCrypt use capacity: med (pinned to 4 GB nodes). Nebula Sync uses capacity: min (it runs for seconds once a day, perfect for the 2 GB node).

Helm

Helm is the package manager for Kubernetes (think apt or brew) but for deploying applications to a cluster. We use it to install the NFS provisioner and Headlamp. Install it on the head node:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh

Verify the install:

helm version

That's all you need. Helm reads chart definitions from repositories and we'll add the relevant repos as needed.

I did not use Helm to deploy Pi-hole. The "why not" is explained later.

NFS Storage

The head node's SSD is the cluster's persistent storage backend. The NFS provisioner creates subdirectories on the SSD automatically when PersistentVolumeClaims are created. As a result, we do not need any manual directory management.

On the head node (assuming you have mounted the SSD and your /etc/fstab is set):

sudo mkdir -p /mnt/ssd/k3s-common
sudo chmod 777 /mnt/ssd/k3s-common

echo '/mnt/ssd/k3s-common xxx.xxx.xxx.0/24(rw,sync,no_subtree_check,no_root_squash)' \
  | sudo tee -a /etc/exports

sudo exportfs -ra
sudo systemctl restart nfs-kernel-server

Install the NFS subdir provisioner via Helm:

helm repo add nfs-subdir-external-provisioner \
  https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update

helm install nfs-provisioner \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set nfs.server=xxx.xxx.xxx.211 \
  --set nfs.path=/mnt/ssd/k3s-common \
  --set storageClass.name=cluster-nfs-client \
  --set storageClass.reclaimPolicy=Retain \
  -n kube-system

This creates a StorageClass named cluster-nfs-client. Any PVC (PersistentVolumeClaim) that references it gets its own subdirectory on the SSD, provisioned automatically.

Here is one workload-specific patch that saves a lot of pain later: Pi-hole uses SQLite internally, so NFS mount behavior matters. I use explicit mount options here to avoid stale cache and locking weirdness (the kind that leads to mysterious database failures and guaranteed hair loss).

kubectl patch storageclass cluster-nfs-client -p \
  '{"mountOptions": ["nfsvers=4.1","soft","noac"]}'

Apply this now and wait for a future part to know exactly why, in painful detail.

---

MetalLB

Bare metal Kubernetes has no cloud load balancer. MetalLB implements the LoadBalancer service type using L2 ARP advertisement - when a service needs an external IP, MetalLB picks a node to announce that IP via ARP, and your router learns to send traffic for that IP to that node. In L2 mode, MetalLB responds to ARP requests on your network - when a device asks "who has xxx.xxx.xxx.232?", MetalLB answers on behalf of the node currently owning that IP, making it look just like any other device on your network. If the node goes down, MetalLB re-advertises from another node automatically.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.3/config/manifests/metallb-native.yaml

Configure an IP pool outside your DHCP range so the router never assigns these addresses to devices:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ip-space
  namespace: metallb-system
spec:
  addresses:
    - xxx.xxx.xxx.230-xxx.xxx.xxx.240
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-adv
  namespace: metallb-system
spec:
  ipAddressPools:
    - ip-space

VIPs allocated in this cluster:

IP	Service
xxx.xxx.xxx.230	Traefik ingress controller
xxx.xxx.xxx.231	Headlamp (Kubernetes dashboard)
xxx.xxx.xxx.232	Pi-hole DNS

---

Headlamp

Headlamp is an open-source Kubernetes dashboard and I found it quite useful for managing my cluster. You can use it to watch pod status, read logs, and look into resources without cryptic kubectl commands. It is not part of the DNS stack but once you have it, you wonder how you managed without it; a must-have nice-to-have.

helm repo add headlamp https://headlamp-k8s.github.io/headlamp/
helm repo update
helm install headlamp headlamp/headlamp \
  --namespace headlamp \
  --create-namespace

Expose it with a MetalLB LoadBalancer service:

apiVersion: v1
kind: Service
metadata:
  name: headlamp-lb
  namespace: headlamp
  annotations:
    metallb.io/loadBalancerIPs: "xxx.xxx.xxx.231"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: headlamp
  ports:
    - port: 80
      targetPort: 4466

Once running, it is available at http://xxx.xxx.xxx.231 from any device on the main network. You will need an authentication token to get into Headlamp - see the documentation for details.

---

Where We Stand

The cluster is now running with all 11 nodes joined, labeled by capacity, with cgroups enabled and swap disabled. NFS storage is provisioning PVCs automatically. MetalLB is managing a pool of IPs. Headlamp is shining bright lights into the cluster.

The foundation is in place, but nothing about so far it is DNS-specific. This setup could run any workload (not any, but be reasonable!).

The next part covers the network design: two physically separated networks, traffic routed between them by a Firewalla Gold Plus, and the supremely annoying problem of giving IoT devices access to a DNS server they cannot normally reach. Plus, do this without compromising their isolation from the rest of the network.

---

Next: Network Design, Firewalla, and IoT Isolation

---

Cover image generated by AI. The fortress may be fake, but the over-engineering is very real.

All YAML files referenced in this series are available in the companion GitHub repository.