MyNewsDesk is a one stop solution for PR and communication, with content creation, analysis and insights, and media monitoring. They are a typical midsized company – 150 employees, 20 developers. They publish around 11000 stories per month, 5 million emails, 6 million page views. MyNewsDesk runs a monolithic Ruby on Rails applications, with React and graphql database.
Until this year, they outsourced most of their devops support to the Heroku platform. They recently embarked on a project to switch platforms. The drivers for the change were GDPR compliance; a feeling they may have outgrown Heroku as MyNewsDesk scaled (having run into some security and development issues); and of course they were looking to see if there was a way to reduce costs.
So what were the alternatives?
They first investigated startups that delivered EU based cloud platforms with GDPR comliance, but these startups were new, and so lacked a track record, and consequently there was no trust for a mission critical application.
The team had zero experience in Kubernetes, but had the idea that maybe Kubernetes was viable for medium sized companies, not just enterprises. After a bit more research, they summarized their feelings about Kubernetes:
Pros:
- Kubernetes is trending to be the de facto deployment platform.
- Open source operators are available for managing HA databases.
- Stronger end game – can customize more for their needs.
Cons:
- The team had zero experience with Kubernetes
- Steep learning curve, and Kubernetes has a scary reputation
- Would have to build an in house platform
However, after getting Kubernetes to work for a hobby application on Nooks at home, they explored viability with a simple proof of concept, replicating the MyNewsDesk site on Kubernetes.
The team elected to use Hetzner for bare metal hosting, as they seemed to provide the highest performance per $.
Talos Linux emerged as the preferred operating system for Kubernetes, for a variety of reasons:
- The team was impressed by how Talos Linux was conceived and its seamless integration with Kubernetes.
- Talos provided a declarative, patchable, and stageable configuration management experience, allowing the team to focus on the Kubernetes part. “We were looking for something to get out of our way.”
- Talos Linux, being a minimal, Kubernetes specific Operating System, made updates simple. “With Ubuntu, you ssh in, you have 150 packages to update, you don’t know what they are, and what will happen when you upgrade, etc. Talos made the updates simple. No package updates to worry about.”
- The team also found that there was an exceptional community Slack. The community helped the team figure out how configuration management could work to make it declarative, patchable and stageable
- Zero headache Kubernetes upgrades were also fantastic.
The cons of Talos Linux was that it’s another learning curve: having no shell felt so strange after being used to normal Linux, so it took some getting used to. It’s also early days for Talos: it is not yet that widely adopted, but as kind of an early adopter of Rails, this is something that is exciting.
The team got Kubernetes running on Hetzner bare metal nodes, with Talos Linux deploying Kubernetes. They struggled a bit with ArgoCD, due to their lack of experience, and had to evaluate 8 different persistent storage solutions, test three, and settled on one. But, after a few weeks, they got the MyNewsDesk application running on the proof-of-concept environment.
For the proof-of-concept, they could compare performance between the two environments, each with the same database and application. On Heroku, they were running 8 x performance L Dynes, and Heroku Postgres premium 5.
On Hetzner, they ran 2 x AX101 servers for web and 2 for databases, with 128GB of RAM. They used Open EBS Dynamic LocalPV provisioner – the only use case for persistent data they had is databases – so they can use DB level replication, and simplify the storage requirements.
They tested latency with a single connection, using one of their busy endpoints: Average Latency went from 205 on Heroku to 134ms on the Talos Kubernetes running on Hetzner. More significantly, the latency of 99% of requests went from 655 to 226ms.
Throughput went from 163 requests/sec on Heroku, with 99% being less than 1.6 seconds, to bare metal providing 261 req/sec, with 99% less than 421 ms.
They also tested sustained load, by replaying seven days of production requests overnight. They achieved 460 requests/second on the Talos deployment, which is 10 x peak production.
Costs for the proof-of-concept setups:
- Heroku: $7650 per month
- Hetzner: $520 per month
This clearly showed that running their application on Kubernetes on bare metal with Talos Linux was viable!
So, they ordered AMD servers, and migrated to bare metal.
To give an idea of scale, the staging cluster has seven of the AX 41 servers and the production cluster is running three of the AX 41’s for the control plane, and then we have five for workers and five for databases.
Looking at the developer experience of actually using this platform, the goal was to make it at least as good as Heroku. The principles guiding the design of the developer experience was to hide Kubernetes from the developers. This doesn’t mean that they don’t have access if they want it, but we didn’t want any developer to have to learn about Kubernetes and YAML and all the complexities. We developed our own command line tool to wrap Kubernetes, that interfaces with GitHub.
We ended up delivering a substantially better developer experience – just one example, deploys went from 5 minutes on Heroku to 10 seconds.
How did the live deployment go?
