Creating a bare-metal cluster from scratch

In the previous section, we looked at running Kubernetes on cloud providers. This is the dominant deployment story for Kubernetes. But there are strong use cases for running Kubernetes on bare metal, such as Kubernetes on the edge. We don’t focus here on hosted versus on-premise. This is yet another dimension. If you already manage a lot of servers on-premise, you are in the best position to decide.

Use cases for bare metal

Bare-metal clusters are a bear to deal with, especially if you manage them yourself. There are companies that provide commercial support for bare-metal Kubernetes clusters, such as Platform 9, but the offerings are not mature yet. A solid open-source option is Kubespray, which can deploy industrial-strength Kubernetes clusters on bare metal, AWS, GCE, Azure, and OpenStack.

Here are some use cases where it makes sense:

• Price: If you already manage large-scale bare-metal clusters, it may be much cheaper to run Kubernetes clusters on your physical infrastructure
• Low network latency: If you must have low latency between your nodes, then the VM overhead might be too much
• Regulatory requirements: If you must comply with regulations, you may not be allowed to use cloud providers
• You want total control over hardware: Cloud providers give you many options, but you may have special needs

When should you consider creating a bare-metal cluster?

The complexities of creating a cluster from scratch are significant. A Kubernetes cluster is not a trivial beast. There is a lot of documentation on the web on how to set up bare-metal clusters, but as the whole ecosystem moves forward, many of these guides get out of date quickly.

You should consider going down this route if you have the operational capability to troubleshoot problems at every level of the stack. Most of the problems will probably be networking-related, but filesystems and storage drivers can bite you too, as well as general incompatibilities and version mismatches between components such as Kubernetes itself, Docker (or other runtimes, if you use them), images, your OS, your OS kernel, and the various addons and tools you use. If you opt for using VMs on top of bare metal, then you add another layer of complexity.

Understanding the process

There is a lot to do. Here is a list of some of the concerns you’ll have to address:

• Implementing your own cloud-provider interface or sidestepping it
• Choosing a networking model and how to implement it (CNI plugin, direct compile)
• Whether or not to use network policies
• Selecting images for system components
• The security model and SSL certificates
• Admin credentials
• Templates for components such as API Server, replication controller, and scheduler
• Cluster services: DNS, logging, monitoring, and GUI

I recommend the following guide from the Kubernetes site to get a deeper understanding of what it takes to create a HA cluster from scratch using kubeadm:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

Using the Cluster API for managing bare-metal clusters

The Cluster API (AKA CAPI) is a Kubernetes sub-project for managing Kubernetes clusters at scale. It uses kubeadm for provisioning. It can provision and manage Kubernetes clusters in any environment using providers. At work, we use it to manage multiple clusters in the cloud. But, it has multiple providers for bare-metal clusters:

• MAAS
• Equinix metal
• Metal3
• Cidero

See https://cluster-api.sigs.k8s.io.

Using virtual private cloud infrastructure

If your use case falls under the bare-metal use cases but you don’t have the necessary skilled manpower or the inclination to deal with the infrastructure challenges of bare metal, you have the option to use a private cloud such as OpenStack. If you want to aim a little higher in the abstraction ladder, then Mirantis offers a cloud platform built on top of OpenStack and Kubernetes.

Let’s review a few more tools for building Kubernetes clusters on bare metal. Some of these tools support OpenStack as well.

Building your own cluster with Kubespray

Kubespray is a project for deploying production-ready highly available Kubernetes clusters. It uses Ansible and can deploy Kubernetes on a large number of targets such as:

• AWS
• GCE
• Azure
• OpenStack
• vSphere
• Equinix metal
• Oracle Cloud Infrastructure (Experimental)

It is also used to deploy Kubernetes clusters on plain bare-metal machines.

It is highly customizable and supports multiple operating systems for the nodes, multiple CNI plugins for networking, and multiple container runtimes.

If you want to test it locally, it can deploy to a multi-node vagrant setup too. If you’re an Ansible fan, Kubespray may be a great choice for you.

See https://kubespray.io.

Building your cluster with Rancher RKE

Rancher Kubernetes Engine (RKE) is a friendly Kubernetes installer that can install Kubernetes on bare metal as well as virtualized servers. RKE aims to address the complexity of installing Kubernetes. It is open source and has great documentation. Check it out here: http://rancher.com/docs/rke/v0.1.x/en/.

Running managed Kubernetes on bare metal or VMs

The cloud providers didn’t want to confine themselves to their own cloud only. They all offer multi-cloud and hybrid solutions where you can control Kubernetes clusters on multiple clouds as well as use their managed control plane on VMs anywhere.

GKE Anthos

Anthos is a comprehensive managed platform that facilitates the deployment of applications, encompassing both traditional and cloud-native environments. It empowers you to construct and oversee global fleets of applications while ensuring operational consistency across them.

EKS Anywhere

Amazon EKS Anywhere presents a fresh deployment alternative for Amazon EKS that enables you to establish and manage Kubernetes clusters on your infrastructure with AWS support. It grants you the flexibility to run Amazon EKS Anywhere on your own on-premises infrastructure, utilizing VMware vSphere, as well as bare metal environments.

AKS Arc

Azure Arc encompasses a collection of technologies that extend Azure’s security and cloud-native services to hybrid and multi-cloud environments. It empowers you to safeguard and manage your infrastructure and applications across various locations while providing familiar tools and services to accelerate the development of cloud-native apps. These applications can then be deployed on any Kubernetes platform.

In this section, we covered creating bare-metal Kubernetes clusters, which gives you total control, but is highly complicated, and requires a tremendous amount of effort and knowledge. Luckily, there are multiple tools, projects, and frameworks to assist you.