Homelab update - k3s and lessons learned
Back in April I wrote about planning a homelab using Raspberry Pis and Kubernetes. Well, it's up and running! Here's a quick catch-up on what I've done and what I've learned so far.
Raspberry PIs have gotten relatively expensive
With the increased popularity of AI, I saw this "AI inflation" directly while shopping around for the Raspberry Pis that I would use in my cluster. I couldn't justify opting for the 16GB memory model since it was over twice the price compared to MSRP! So I ended up opting for the 8GB model, which still offered double the memory compared to my RPI 4 that was still chugging along in the garage. That said, this was one of a few instances where the best option for AI workloads was clearly overpriced, even with a smaller usecase/runtime such as an RPI 5, this price point was rough.
The RPI 4 I already owned is still the "main node" for two reasons: I can't use it for larger models or a lot of processes, but it was available first, so I could set it up and get it ready and running before the others arrived. Either way the plan was to expand the homelab with a larger model/device later, and just utilize the newer RPI5s as the main nodes to get a feel for things.
The goal for the hardware was never firm, so I could tear it all down and build a new one based on these models. The only one that would “hurt” is tearing down the RPI 4, as I use it as the main network hub and interaction point. I also still spent a good chunk of change building out this smaller network, with the purchase of 3 RPI5s, and the extras to support the devices. I still need to clean out the part of the wall where I’m going to mount these.
With the prices the way they are, I wouldn’t consider this very affordable. Its more affordable than running or buying higher end hardware, such as the how I considered just buying a single framework desktop, but since they are separate machines that a perk in itself, even if it is something of an artificial complexity.
The kubernetes cluster
Beyond the hardware components, the main goal of having all this hardware to play around with was to get more familiar with kubernetes and see how such a smaller scale system could potentially run smaller scale AI workloads. I had to try to run models, but the user experience just wasn’t there—though it was worth trying!
The Kubernetes cluster was largely built using GitHub Copilot to spend those monthly credits, with minimal oversight from me besides getting the general direction. The network consists of four nodes, but only three can be used for various services. I let Copilot make the decision to define the three nodes by names (Alpha, Beta, Charlie) so it's easier to identify which RPI is running what at any given time.
I thought about setting up Tailscale and even having a droplet somewhere serve as a “front-door” entry point from the public internet, but I haven’t found the need or motivation yet to expose the stack to the open web, even through such a setup. This might come later if I decide I want to do that, but lately my focus has been on other stuff, which will be another blog post later!
Otherwise, I was able to play around with Ollama running on each of the nodes to run smaller models. I used k9s to set up the initial cluster, a lot of Copilot and Gemini to get an idea of how to connect all the documentation, and SSHing into all the boxes.
Finally I mixed in some basic monitoring with uptime kuma, and some baseline services such as gitea. I haven’t really leveraged the network beyond this and trying to spam ollama, but its a start!
Lessons learned and future
So, one of the main lessons I learned is that Kubernetes isn't really kind to a beginner. I never was a fan of yaml files, but at least this time I had copilot trying things out and helping me do stuff without trying to memorize everything myself. Still, making sure Copilot wasn’t making errors in the setup was always a little annoying. I also think what I’m getting out of the stack compared to what I’d need to put in is very questionably worth it since the cluster is so small.
I had one goal of using the RPI network to run Opencode, but this just isn’t possible with this hardware. The RPIs are just too slow and not able to do much more than smaller model work; Opencode's system prompt must require too much processing power. I could have tested this all locally on my laptop, but I really wanted to play around with new tech so I’d consider it an expensive lesson that still proved useful.
Because of these issues I think expanding the network with more nodes and more AI focused work is probably off the table. I’ll probably be redirecting future funds in another direction, which I’ll be writing about later. In the meantime, it was a great learning experience for something I am not super familiar with.
Until next time, keep learning, keep building and keep asking questions!