Automatic Cluster Scaling with Omni

Kubernetes autoscaling isn’t a new feature. The official autoscaler has been used in many environments to dynamically add and remove nodes from a cluster.

This behavior is useful to save money if your workloads automatically scale up and down and you are charged dynamically per node—like in a cloud provider. Dynamic scaling is also useful for automatic cluster provisioning and to temporarily scale resources to handle large events (e.g. cloud bursting). Omni handles automatic cluster scaling through Machine Classes. We’ll show you how to do it so you’ll be prepared when you need it.

If you want a video walk through showing how this works check out the video below. Thanks to Oxide for lending us resources to show how fast you can scale on-prem Kubernetes clusters.

Many environments don’t require dynamically scaling workloads and on-prem environments don’t usually charge dynamically for compute resources. So we’ll also review how you can manually scale clusters to match process and workload requirements at the end of this article.

Create a Machine Class

The first step to automatic scaling is to define what types of machines you want to scale. Machine Classes match labels on nodes with a variety of operators. You can have complex Machine Classes such as every node in a specific data center, running on VMware, with more than 8GiB of memory.

Classes can be created by clicking on existing labels or manually typing the conditions as needed. They also can be declared in a YAML document and sent to the the Omni API via omnictl. For example, this creates a Machine Class called “metal” that matches all machines using the platform: metal.

metadata:
    namespace: default
    type: MachineClasses.omni.sidero.dev
    id: metal
spec:
    matchlabels:
        - omni.sidero.dev/platform = metal
    autoprovision: null

Then you can apply that to Omni with

omnictl apply -f metal.mcs.yaml

Create Cluster with Machine Sets

You can create a cluster in the Omni UI and you will automatically have two Machine Sets. One for control plane nodes and one for worker nodes.

You can add more machines sets or use a Machine Class to automatically add machines to your Machine Set. A Machine Class can be thought of as a pool of compute resources to add to the cluster.

For example, maybe you want your control plane nodes to run on OpenStack and your worker nodes to be provisioned with bare metal. You can create a Machine Class for each type and then you don’t have to manually pick which machines go in the cluster. Just select how many machines from which Machine Class.

Machine Sets are like “node group” in some Kubernetes services. It’s a generic pool of systems that can be controlled together. Machine Sets don’t have to pull from a Machine Class and don’t have to be the same size or type of machine. They don’t even have to be in the same region. As long as they’re connected to Omni they can be used in a cluster. You may want to check out KubeSpan to create clusters that span networks.

Machine Sets can be helpful for patching a group of systems, but they don’t have to be used this way. You can see in the picture below that a cluster patch can target the entire cluster, only the control plane nodes, the “Workers: default” Machine Set, or individual nodes in the cluster.

You can also provision a cluster with machine classes using a cluster template. Cluster templates make it easier to deploy, modify, and track changes. A cluster template will look similar to this.

kind: Cluster
name: oxide
kubernetes:
  version: v1.32.0
talos:
  version: v1.9.4
---
kind: ControlPlane
machineClass:
  name: vmware
  size: 3
---
kind: Workers
machineClass:
  name: oxide
  size: 15

In this example we create a cluster with 3 control plane nodes from the “vmware” Machine Class and 15 workers from the “oxide” Machine Class. It will automatically install Kubernetes 1.32.0 and Talos 1.9.4 once the machines are ready.

You can apply a cluster template to the Omni API with:

omnictl cluster template sync -f oxide.cluster.yaml

Now you just need to connect machines to Omni. The cluster will automatically scale with machines that match the Machine Classes up to the limit specified.

If you find you need to scale your cluster once it is already provisioned, all you need do is change size: 15 of the Worker machine class to say, size:20, rerun the same omnictl command, and Omni will automatically scale up the worker pool to 20 machines if and when they’re available.

There is also a special Unlimited size which will match all machines in a Machine Class. This is useful for Machine Classes that narrowly match a specific type of machine dedicated to a specific cluster. Any time a machine that matches the Machine Class is powered on and registers itself with Omni, it will automatically be added to the cluster—effectively allowing you to move cluster scaling logic into the process that powers on machines, like AWS auto scaling groups.

Manual Scaling

Scaling automatically isn’t always desired. Sometimes machines are specifically allocated to a team, application, or workload and need to be managed manually.

In that case you can still scale a cluster by clicking Cluster Scaling in the cluster overview.

And selecting the machine you want to add or remove from the cluster. You can create new Machine Sets or use existing Machine Sets.

Conclusion

Just because you’re running on-prem doesn’t mean you shouldn’t be able to dynamically scale your clusters. Omni clusters can scale dynamically with the cluster autoscaler but they’re not limited to using it. If you add compute resources from any infrastructure provider Omni can add it to a cluster.

If you want to get started with the best on-prem and edge Kubernetes cluster management sign up for an Omni 2 week free trial and get started.