Kubernetes is a great place to run your applications. The abstractions and packaging has proven to be portable across multiple environments and use cases. But Kubernetes relies on underlying infrastructure to run containers. That infrastructure ranges from bare metal to cloud and is provisioned by everything from manually installing operating systems to terraform.
Omni is an infrastructure agnostic platform for deploying and managing Kubernetes. If you can boot Talos Linux, you can use Omni to create a Kubernetes cluster. But creating VMs, PXE booting hardware, or mounting ISOs has been your responsibility, until now.
Today, we’re introducing Omni Infrastructure Providers to automatically provision and manage compute infrastructure. Manual provisioning workflows still exist, but we think you’ll love the automatic approach with providers even more.
What is an Infrastructure Provider
Providers connect Omni to your infrastructure APIs. That infrastructure can be on-prem behind your firewall, or in a public cloud. The machines that are provisioned can be bare metal managed via IPMI/Redfish, or they can be dynamic, virtualized infrastructure from a platform like VMware. Any way that allows you to programmatically manage compute can be defined as an Omni Provider.
Providers work similarly to Cluster API Providers in the way they can create infrastructure resources, but Omni Infrastructure Providers are different in a few key ways:
- Native bare metal support
- No Kubernetes requirement
- Minimal machine management
- Mix-and-match providers
Let’s look at what those mean and then we’ll show you how to get started.
Native Bare Metal Support
Cluster API providers assume all machines are replaceable. You should never do an in-place upgrade of an OS or Kubernetes because “immutable is the way!”
That is great in theory, but in reality, people may not have spare computers to cycle in and out for upgrade cycles—especially if they are running expensive GPUs, or have huge amounts of local storage that they do not want to have to replicate, just for an upgrade.
Sidero knows this, and we focus a lot of our tooling around bare metal Kubernetes management. We know some machines should be reinstalled in-place, and there may be no dynamic way to get more machines because they’re made of atoms, not bits. Omni was built with this idea in mind and we make sure bare metal is a first-class consideration for all of our tooling.
This also means provisioning needs to happen lower in the stack. Many CAPI providers assume they can make an API call and then execute code on an OS. The Omni Bare Metal Provider knows that physical machines require network access, and an OS needs to be installed first.
Omni uses Talos Linux for machines which is the only single-purpose, Kubernetes distro built to run anywhere. Because of its API management and limited scope, it makes managing servers easier than general purpose Linux distributions that need to be configured and hardened to work with Kubernetes.
No Kubernetes Requirement
Cluster API leans heavily into “everything is a CRD” in a way that becomes difficult to manage. If you have an edge environment with minimal resources, the first step to use Cluster API is to provision a Kubernetes management cluster. That’s no easy feat. Now you have to keep that cluster updated and it becomes a critical component for any clusters where you want to run applications.
Omni is deployed as a single container. It can run inside a Kubernetes cluster, but it’s perfectly fine deploying as a container on a Linux VM. Providers are the same way. Because they store their state inside Omni and connect via the gRPC API, they don’t have to run physically close to the Omni instance.
This architecture allows you to use the Omni SaaS, but still have a provider installed in your network where it can provision infrastructure that is not available outside your firewall. It also means you don’t need to worry about managing extra Kubernetes clusters or exposing internal APIs to external Kubernetes controllers.
If you have 3 datacenters and you want to provision bare metal servers in each one you can deploy the Omni bare metal infrastructure provider in each datacenter, PXE boot the machines, and Omni takes care of powering the machines on and off, installing the OS, and provisioning clusters.
Minimal Machine Management
Infrastructure providers are designed to create and destroy machines, and boot Talos Linux: that’s it. This simplifies what providers can do and should make it easier for you to write your own. Because Omni doesn’t care about where the machines come from, all configuration and identity is part of the operating system, not the hypervisor.
In cloud providers, machine identity is typically handled at the hypervisor layer. This means you need to set up IAM rules and cloud-init before you provision machines. With Omni, this is all handled in Talos Linux which means bootstrapping machines is greatly simplified and portable.
Omni providers don’t provision managed Kubernetes services like EKS. Instead, Omni provides automation and management that makes creating and operating Kubernetes clusters as easy as a managed service. This eliminates the complexity of trying to deploy managed and un-managed clusters with the same tools. Cluster API tools like clusterctl and clusterawsadm are unnecessary because of the simplified scope of machine management.
Mix-and-match Providers
One thing that many of our customers do is span their Kubernetes clusters over multiple infrastructure types and platforms. Some customers like to put their control plane nodes in a VM environment where they can pack more of them on fewer hardware nodes. Other customers temporarily burst their bare metal infrastructure into a cloud provider to handle increased load.
With tools like Cluster API this can be difficult because providers aren’t designed to work together. With raw infrastructure provisioning tools like terraform it’s up to the engineer to provision and connect infrastructure together.
With Omni you can manually deploy Talos to a Raspberry Pi, automatically provision bare metal servers with the provider, and create VMs in the cloud. You connect them all using SideroLink and KubeSpan and deploy them into a single cluster. Omni and Talos are agnostic to where they’re running and network connectivity is built into the OS to make it seamless to run from anywhere.
Getting Started
Because providers are designed to be stateless connectors between Omni and your infrastructure, they’re built as single binaries and packaged in containers. If you know how to run a container, you can deploy a provider.
The only prerequisites to running providers is to have an Omni instance available and a service account created for the provider to communicate to the API. You can use the Sidero managed Omni SaaS or you can use Omni self-hosted and run it in your environment.
To create a service account in Omni you can use omnictl
omnictl serviceaccount create
--use-user-role=false
--role=InfraProvider infra-provider:bare-metalThis will show an Omni endpoint and service account key you’ll need to run the provider.
Some providers don’t need credentials to create machines. For example, the bare metal provider provisions infrastructure via PXE booting machines. The bare metal provider is known as a “static” infrastructure provider because machines are added to the pool of Omni resources and can be added to clusters as needed. When a bare metal machine isn’t part of a cluster, it will be shut down to reduce power and cooling consumption.
To deploy the bare metal provider you can run the following command:
docker run -d --name=omni-bare-metal-infra-provider
--restart=always
--network host
-e $OMNI_ENDPOINT
-e $OMNI_SERVICE_ACCOUNT_KEY
ghcr.io/siderolabs/omni-infra-provider-bare-metal:v0.1.0
--api-advertise-address=172.16.0.42Make sure to replace your $OMNI_ENDPOINT and $OMNI_SERVICE_ACCOUNT_KEY with the required values. The --api-advertise-address option tells the provider what machine IP address you’re running the provider on which is necessary for PXE booting.
That’s all you need to do. Now you can PXE boot a machine on the same network and the provider will discover the machine and connect it to Omni.
Machines may have been PXE booted by mistake, so we added an extra step requiring a machine to have been “accepted” before it can be used in a cluster. These machines will show up as “Pending” in the machines list until you have accepted or rejected them.
You can accept them into the Omni inventory or you can reject them and the Omni provider will not PXE boot them in the future, and their current disk state will not have been altered.
Warning: Machines that are PXE booted and accepted in Omni will have their local disks wiped
After a machine is accepted, all machine maintenance operations are the same as self-managed machines. The only difference is Omni has additional information about the machines such as their current power state.
Dynamic Infrastructure Providers
Dynamic Infrastructure Providers create and manage software defined resources such as VMs. Dynamic providers work differently than static providers because dynamic providers will create machines on-demand when a cluster is created or expanded.
For a dynamic infrastructure provider you will provide a service account for Omni and API credentials for where the infrastructure will be provisioned. For a provider like the KubeVirt provider you would provide a Kubeconfig file like so:
docker run -it -d -v ./kubeconfig:/kubeconfig
-e OMNI_ENDPOINT=$OMNI_ENDPOINT
-e $OMNI_SERVICE_ACCOUNT_KEY
ghcr.io/siderolabs/omni-infra-provider-kubevirt
--kubeconfig /kubeconfigNo VMs will be created until you request them in the cluster creation workflow. To see a video that shows the entire flow of deploying providers and creating clusters you can check out the video here.
Conclusion
Automating infrastructure has never been easier. A single container can bridge the connection between Omni and your infrastructure to connect machines statically with bare metal or dynamically with VMs. This gives you the freedom to run your Kubernetes clusters on whatever infrastructure you have available and even move between them!
Get started with Omni today, and let us know what providers you’d like to see in the future.