Talos Linux is an API driven distribution for running Kubernetes. It takes great ideas behind immutable operating systems a step further. It isn’t a container operating system, it’s a Kubernetes operating system. The difference is, Talos is built only to run in Kubernetes. Other minimal distributions, like Bottlerocket, add complexity by supporting multiple workloads and orchestrators.
By focusing Talos only on Kubernetes and a declarative API it will greatly reduce the toil required for running a Kubernetes cluster. It can reduce it so much there are only 12 binaries in Talos.
Bottlerocket is a container focused Linux from Amazon Web Services. It was launched in 2020 to compete with Talos. The main goals of Bottlerocket—according to their website—are minimal, safe updates, and security focused.
Let’s take a look how Talos Linux and Bottlerocket compare so you can better understand how they’re similar and where they’re different.
Safe updates
Bottlerocket uses A/B partitions to download image updates and reboot into the new partition. While this doesn’t always guarantee it’ll work, it can guarantee the option to roll back to a working partition.
This is great! Talos does a similar thing with booting and switching to the next squashfs image.
Bottlerocket is configured via an API which helps migrate configuration between different version of the OS. However, Bottlerocket has fewer than 200 configuration options and more than 50 of those are AWS specific. None of them are for debugging.
Having an API for configuration sounds like Talos, but in reality there are minimal settings available. If you needed to set up bonded networks or gather CPU info there would be no endpoint to call. Talos works in more environments than Bottlerocket and has thousand of configuration options available through the declarative API.
If you need to change Bottlerocket network configuration you can set a hostname, modify /etc/hosts, and change proxy settings. That’s it. If you’ve ever run a physical server or even a VM in a non-AWS environment you’ll know that’s not enough.
Any OS settings that are not available via the API requires you to build Bottlerocket from source. You would be responsible for updates, security, and distribution of your own variant. At Sidero, we believe if you have to build Talos from scratch to configure something it’s a bug.
We provide multiple ways to configure the OS via the API, system extensions to add drivers and runtimes, and overlays for pre-install configurations (more on this soon).
Minimal
Bottlerocket claims to be “minimal” but what does that mean? According to their website that means they don’t have a package manager and they build “variants” to match the version of Bottlerocket you need with the environment and orchestrator you’re using.
Not including a package manager is not new, nor is it minimal. Bottlerocket has more than 250 binaries in it’s $PATH. While a binary count isn’t a direct comparison of how minimal it can be used as an indicator of how minimal it is. Talos has 12 binaries.
Variants might minimize what goes into an OS, but it greatly increases the amount of work you need to do to figure out which image you need to download. Bottlerocket supports EKS and ECS which are competing AWS container orchestrators. They support running Bottlerocket on EC2, VMware, and on metal. However, I never gotten Bottlerocket to boot on any of my test machines and the only option was to build it from scratch to include necessary drivers.
Bottlerocket also provides an NVIDIA variant, but only the driver version they pick. When and how they update the driver is up to them. If you need a different version of the driver you have to build from source. I wrote the AWS blog post when this launched and I believed the way it was managed was a bad idea then and I still believe it’s a bad idea today.
Talos makes these things composable. You can select any supported version of Kubernetes you need with any supported version of Talos. Your OS and Kubernetes versions should not be tightly coupled. Your drivers, extensions, and configuration should also be composable.
Talos is so simple you can run it directly in Docker
Talos supports 12 different cloud providers, a variety of SBCs—such as Raspberry Pi, and supports arm and amd64 architectures. Extensions let you combine different options while still keeping the OS minimal and declarative. If you need a USB modem and want gvisor, no problem. Go to factory.talos.dev and select the options you need.
Security
When Bottlerocket says it’s security focused they mean that in the “it’s hard to use” definition. I’ve been demoing and talking about Bottlerocket for years and it has always been difficult troubleshoot and show how things work.
The main reasons I say this is because of these three design decisions
- Session manager
- Maintenance containers
- Secure Boot
Session manager
There is no SSH on Bottlerocket—that’s great. But there is a proprietary protocol for getting an interactive terminal with host level access—that’s bad. Just because you disable SSH doesn’t mean you should replace it with websockets.
Bottlerocket uses Session Manager which is a websockets based tunnel that contacts an AWS endpoint, which sends a message to an instance and forces the session manager daemon to make an outbound connection to the endpoint and your terminal is connected. This turns a hardened and battle tested protocol like SSH into a single vendor, closed source, back door that you have to pay money to use.
That’s not more secure. It’s more obscure.
Container runtimes
Bottlerocket provides 2 container runtimes (containerd x2 for Kubernetes). They do this for workload vs system isolation. Talos does this too! System level containers like the Kubelet don’t need to be run by the same containerd daemon.
But with Bottlerocket they also split how you can manage the system by a “control” container and an “admin” container. This means that if you want to contact the API that is used to configure the system you have to get a shell on the server in the “control” container, switch to the system container runtime “admin” container, and then call the API on localhost. It’s a bad experience.
Bottlerocket’s API endpoint is unauthenticated and “protected” because it’s not exposed to an external network interface. It also has a CLI—apiclient—and bash script sheltie to switch between the control and admin containers. These are not API driven nor are they automation friendly.
If you forget to enable the admin container at provisioning time then you’ll need to update the machine metadata and reboot. This will likely eliminate any possibility of debugging a problem unless you leave the admin container always enabled (which they don’t recommend).
If you ever wanted to look at system logs on a Bottlerocket node you’re going to have to leave the admin container enabled. There’s no way to send system logs to a central logging system. Talos not only allows for this configuration in the MachineConfig it also gives you log streams over wireguard by default with Omni.
Talos also has a TUI console dashboard for local and remote monitoring
Talos has talosctl so you can manage your systems from a separate machine. It uses similar principles to Kubernetes to access nodes through an API server (called endpoints) and keeps everything secure with mutual TLS. It makes an API driven Linux distribution easier to use, not harder.
Secure boot
Bottlerocket enables Secure Boot on systems that support a UEFI boot. That’s great! It secures some of the boot process to make sure the early stages are verified and unchanged.
However, Bottlerocket doesn’t do this for NVIDIA or any variants that need to load unsigned kernel modules. That seems odd, because they should sign the kernel modules like Talos does! This validates the kernel and modules that run.
Talos not only supports Secure Boot—even with NVIDIA drivers—it also supports Trusted Boot which is a more thorough capability to protect your OS and workloads from modification. This can give you confidence that your OS and workloads will not be tampered with even if you run Kubernetes in untrusted environments like edge deployments.
Did we mention that Bottlerocket doesn’t encrypt volumes outside of AWS EBS volumes? If you install Bottlerocket on a metal host and pull out the hard drive you can get access to any and all files on the system. This could also mean workloads that have written files to disk—temporary or permanent. Talos can encrypt OS volumes no matter where you run the operating system.
All of this “security” in Bottlerocket and it still uses a general purpose init system with systemd. This means that every CVE that has affected systemd also affects Bottlerocket to some extent.
Next time your looking for a minimal, safe, and secure operating system you should start with talos.dev.