Talos Linux on Proxmox: The Immutable Kubernetes Dream (and its Nightmares)

In one of my previous posts about the state of my home cluster, I briefly touched on why I decided to use Talos Linux. When deciding on how to set up your Kubernetes cluster, you have what feels like infinite choices. There are over 60 available Certified Kubernetes Distributions, some of which are tailored towards corporate use (I am looking at you OpenShift and your what-feels-like-infinite pricing), while others are tailored for single-node development environments.

Then you also have to choose how you install it. Do you want to run it on a virtualized setup, or on bare metal hardware? And if you choose a version that can be installed on multiple distributions, which one do you choose? Should you install K3s onto your Ubuntu or openSUSE server?

There are questions upon questions, yet in my opinion, Talos solves most of them.

On one hand, it requires unlearning almost everything you know about Linux systems administration. Talos is meant to be as lightweight as possible, meaning there is no SSH server you can access, there is no normal package manager, there isn’t even a bash shell you can connect to. Connecting a monitor only gives you some basic information, like the cluster status.

In this post, I will break down exactly why this radical approach is brilliant, where it falls short, and how it stacks up against the alternatives in a Proxmox environment.

How does Talos Linux compare to other distributions?

Talos is a modern OS designed for one specific purpose: running Kubernetes. It strips away all the legacy bloat of a general-purpose Linux distribution.

1. Comparison against standard distributions

Let’s simply compare it to one of the most common Linux distributions out there: Ubuntu. (And don’t worry, there will be a future post about why Ubuntu is awful and why, for most use cases, you should simply use Gentoo). The main reason you should use Talos is the significantly limited attack surface. There are simply far fewer packages installed; Sidero Labs (the developer) claimed only about 12 binaries back in Talos 1.7, and it remains well under 50 today. For a deep dive into exactly which packages are included, I recommend reading their own blog: There are only 12 binaries in Talos Linux.

What you need to know is that any default Linux distribution has much more installed out of the box. If you run an Ubuntu server, you either need to manually uninstall a ton of packages, or you must accept that if SSH gets a critical CVE, your server is affected. Or, as happens over and over again, anything related to systemd (another blog post coming about this topic).

2. The API-Driven “No SSH” Philosophy

As I just mentioned, there is no SSH server installed on Talos Linux. So how do you administrate it, get logs, or update the system?

There is a universally brilliant answer to this named talosctl. It is, of course, open source and available on the official Sidero Labs GitHub.

Instead of needing to SSH into all nodes to upgrade them, you simply run a command that upgrades your nodes over the API: talosctl upgrade --nodes <IP> --image ghcr.io/siderolabs/installer:v<VERSION> As with all K8s distributions, it is recommended to update the control plane nodes one after the other. You can do that manually or with a basic bash script—whatever fits your needs. talosctl also allows you to add as many nodes as you want with a single command, add kernel modules, and much more.

The massive advantage here is that you never manually edit any configuration files live on the server, which means human-error issues are far less likely to happen. On the downside, this means that many standard break-fix troubleshooting methods (like cat /var/log/syslog or lsblk) simply do not exist when using Talos.

3. Immutable and Declarative

3.1 The Declarative Advantage

Another huge advantage of Talos Linux is the fact that it uses an immutable base and a declarative system. Many of you have surely heard of NixOS already, and Talos uses a similar philosophy.

When setting up Talos, you generate configuration files (Machine Configs) to apply to your nodes, specifically tailored for your control plane and worker nodes. The huge advantage is that you only have these YAML files to worry about when modifying your cluster. A few examples of what you can do declaratively include:

• Adding and removing registry mirrors
• Changing network settings and static IPs
• Configuring external storage mounts

3.2 The Immutable Advantage

Compared to how standard Linux distros work, immutable systems don’t constantly write to the root drive. Instead, Talos boots a read-only SquashFS image and runs the OS entirely from RAM (tmpfs).

This means there is no “configuration drift.” Your server cannot slowly break over time because of leftover dependencies or bad updates. Every single time a node reboots, it boots into a perfectly clean, pristine state identical to the last. If an upgrade ever fails, the node automatically rolls back to the previous immutable image. No broken states, no dependency hell.

4. Why Talos may not be for you

So after a lot of praises, you may ask yourself “should I immediately install a Talos cluster???”, so let me outline a few reasons why you may not want to install Talos.

• As already mentioned, there is no SSH or anything like that, so you need to learn to use the API/talosctl and nothing else. No “network is broken, let me quickly try to troubleshoot it”. You simply can’t.
• There is no persistent storage. The filesystem is read only, so whenever you want to save something, you basically need an external server to run some kind of file storage server. So before deploying a database, you want to set up some kind of CSI. This can be csi-nfs, Longhorn and much more. But you are forced to deploy it before running anything that should persist storage in any way. Now you are supposed to do this on an k8s Clusters, but being forced to may be a bit offputting for a newcomer.
• If you need some kind of extra kernel modules (luckily not the case on Proxmox), it is quite different to how it works with something like Ubuntu. Not a big issue nowadays, just something to keep in mind. This as well requires the usage of talosctl.
• Last but not least, this is mainly geared towards the “I want to use the newest tool!” crowd: Talos and Kubernetes are not replacements for Docker or VMs. They are a whole different beast and require much more specific knowledge in my experience. You also get more control and power, but with great power comes great responsibility.

5. Why not OKD?

This is a question I do expect to be in the mind of at least a few people. My reason as for why I chose Talos is quite simple. OKD has a “use my tools” philosophy, whereas Talos has much more of a “bring your own tools” philosophy. This fits better into what I want. Instead of an opinionated “use X to do Y”, I much prefer the simplicity of what Talos offers. Another downside of OKD is that OKD requires much more resources than Talos does. This is partly because OKD is more aimed towards being a development version of OpenShift (or as I like to call it, OKD is the Fedora in K8s), so it is around a similar ressource usage as OpenShift. Another aspect that I don’t need is the fact that Fedora CoreOS still comes with all the tools a normal Linux distro does, so you get a full SSH daemon, you get full systemd and so on.

Or in short: Choosing OKD does force you to use a whole bunch of tools, Talos does only give you the bare minimum and makes you decide what you want to use.

The Proxmox Context

As written a few days ago, I use Proxmox as the base for my Cluster. This actually has quite a simple reason: I want dedicated worker & control plane nodes, without needing to buy dedicated hardware for each (as well as being able to set them easily with however much compute I wanna give each node). With Proxmox, I use 3 hardware nodes and each hosts one control as well as one worker node. Meaning that even if there is a hardware outage on one node, the cluster itself still works.

Conclusion

The conclusion is kinda boring as there is “do what fits your world the best” in the air. I will always say that choosing a minimal system that makes you decide what you want to do is much better than choosing something that forces you to do your tasks in a specific way. The monolithic OSes like Ubuntu with a k8s installation on top are too big and too manual, distributions like OKD force too many tools on you. Based on my experience, Talos is one of the few distributions out there that

• Follows the Unix way (one tool, one task)
• Makes you decide what how you do what you want/need to do

And what does fit better than a tool that gives you freedom than choosing something that locks you down?