Flatcar Linux was recently accepted as a CNCF Incubator project and is the spiritual successor to CoreOS before it was changed into Fedora CoreOS. There’s a long history behind CoreOS we won’t go into here. It was the OS I used to create my first bare-metal Kubernetes cluster in 2016 and it’s gone through some technology and ownership changes since then.
We often get asked what the differences are between Flatcar and Talos Linux so we’ll try to describe some of them here. But first let’s start with how Flatcar describes itself and how Talos compares.
- Container optimized: which says it “only contains those packages needed to run containers” which is surprising to us. Talos only has 12 unique binaries in the system PATH while Flatcar has over 2300. It’s unclear what all of the binaries are used for, but they increase the surface area of risk and update maintenance. Even Amazon Linux 2, a general purpose Linux distro, has fewer binaries than Flatcar.
- Secure and Immutable Filesystem: Flatcar says its OS uses a “read-only filesystem” which is true that the
/usrpath is read only. But many parts of the filesystem are not read only, including/. Talos makes more of the file system read only and even goes further to bind mount only specific files into configuration paths like/etc. - Declaratively Provisioned: Flatcar uses butane for pre-boot configuration and cloud-init for post-boot configuration. While both are formatted as YAML files it doesn’t mean it’s declarative or reproducible. The docs for creating a Kubernetes cluster with Flatcar include downloading 3 files from GitHub, 2 heredoc style text blocks, and a systemd timer (a.k.a. cron job) that executes
kubeadm init. That’s not declarative or immutable. It sounds a lot like a YAML formatted bash script. - Auto Updating (and Rollback): Flatcar was inspired by ChromeOS which has an automatic update service with image based upgrades. It uses 2 partitions to keep an A/B version of the operating system. This means you can more easily (and automatically) roll back if something fails. This type of upgrade is reliable and used in many different operating systems (including Android). Talos Linux also has this update style although Talos doesn’t provide automatic upgrades or subscription channels like Flatcar. If that’s something you’d like from Talos please let us know.
Now let’s look at some other differences you may want to consider.
API driven vs configuration driven
The key benefit of Talos Linux is its grpc API to drive all configuration of the system. This applies to the local machine configuration such as partitions and kubelet, as well as cluster configurations like etcd and certificates.
Because Flatcar only has automated configuration via butane and cloud-init it becomes increasingly difficult to run Flatcar on bare metal where those are not available by default. It’s not that you can’t do it, but it’s the only option.
Talos provides automatic configuration via cloud-init just like Flatcar. But unlike Flatcar which requires you to write a bunch of files to disk to run systemd services or download binaries, Talos’ configuration is actually declarative. The Talos API also lets you configure the machine automatically during boot or declaratively after boot.
Anything you would want to configure on Talos is done via the API with no SSH required, even debugging. On Flatcar as soon as something breaks you have to fall back to traditional system management with SSH (including SSH keys and local users) and installing tools like tcpdump to get data. Talos Linux provides this data via the API with no SSH, user configuration, or installation.
General purpose container vs single purpose Kubernetes
Flatcar is Gentoo based and is intended for container usage. It comes with a container runtime as its primary way to “install” and run software and can be used with Kubernetes or other container based orchestrators. It doesn’t come with traditional package managers like dnf or apt but it does come with a command called toolbox which will run a traditional Linux container with elevated privileges so running dnf is only one step away.
Flatcar also comes with Docker pre-installed, but Kubernetes doesn’t use docker as a container runtime. This is one more example that Flatcar is intended to be used outside of just Kubernetes. This is a great option if that’s what you need.
Talos Linux is only for Kubernetes and makes some assumptions about what you want to do with the machine. It’s not intended to install Talos Linux for arbitrary containers without Kubernetes but with Flatcar that’s fine.
Init systems
Flatcar uses systemd as its init system which has all the flexibility and quirks of systemd. I really love systemd for general purpose Linux. I’ve been using it on all my desktops and laptops for a long time, but I dislike it for single purpose use cases (eg Kubernetes).
Talos has a special purpose init system responsible for the system grpc API and booting the kubelet. That’s about it. It’s a few hundred lines of go (not including imports) and has a single purpose (and limited flexibility) compared to systemd.
Flatcar claims to have a “minimal attack surface” but all of the CVEs that have affected SSH, systemd, and cloud-init in the past also affect Flatcar because it doesn’t go far enough to reduce requirements.
Extensions
Flatcar announced this year that they have adopted systemd-sysext to extend the way you can add drivers and services to Flatcar. This lets you create your own raw images and layer them as part of the file system. We agree this is a good approach which is why we do something very similar with our system extensions.
The difference is with Flatcar you need to “bake” and maintain extensions yourself. You have to integrate them into the system with cloud-init, and you have to configure them via more files on your disk. This integration happens after the system boots and can take 5-10 minutes for it to be available. The Flatcar nvidia.service automates this process for NVIDIA drivers. We don’t think that experience is good enough.
With Talos we run a service called the image factory which lets you layer in whatever drivers and services you need with a few clicks. Extensions are added at build time instead of at run time just like with containers. It is already available and extensions are open source. The image factory has common drivers like NVIDIA and services such as Tailscale. You can also use it from the command line or self-host it if you need to add proprietary or custom extensions. Read more about it in our documentation.
Hardware support
Flatcar has x86 and arm64 builds of their operating system, but it still has limitations on where the system runs and its intended use cases. Flatcar typically assumes you’re running VMs in some way to provide cloud-init and butane configuration. Without those files in place there’s no easy way to automate the configuration.
Because Talos exposes a secure grpc machine API it doesn’t require a cloud-init file to be configured. If you want to configure the machine after it has booted that works just the same. This is ideal for edge and bare metal environments where you may not have the ability to easily mount a metadata or ISO filesystem. This is how our Omni service works by using the API to provision and maintain servers wherever they run.
Talos also works on single board computers by using u-boot. Talos overlays are part of the image factory and can be configured and added to a Talos image the same way as an extension. This lets you run Talos on popular SBCs like the Rasberry Pi, Jetson Nano, or Pine 64 boards.
Flatcar has unofficial support for Raspberry Pis using a custom firmware. While this might be OK for a home lab there’s a lot of other use cases for SBCs and a lot of variety in how they work.
Conclusion
CoreOS was a new way to think about operating systems when it came out in 2013. The primary benefit was safe and automatic updates. But the OS formally known as CoreOS, Flatcar, hasn’t improved much beyond that promise. CoreOS was deprecated in 2020.
Lots of operating systems will give you automated updates. Some of them even make them safe to preform and roll back.
Talos has a different goal. We want to further the goal to make upgrades automatic and safe, but we also want APIs to drive everything we do with infrastructure. From application deployments with Kubernetes all they way down to the Linux kernel.
We wanted a single purpose Linux distributions for Kubernetes. We wanted to remove all of the general purpose configuration and tooling and focus on doing one thing better than anyone else. If you haven’t tried Talos yet, you’re missing out on the future.