In this section, we will create a local single-node cluster using Minikube. Local clusters are the most useful for developers that want quick edit-test-deploy-debug cycles on their machine, before committing their changes. Local clusters are very useful for DevOps and operators that want to play with Kubernetes locally, without concerns about breaking a shared environment. While Kubernetes is typically deployed on Linux in production, many developers work on Windows PCs or Macs. That said, there are not too many differences if you do want to install Minikube on Linux:

Figure 2.1: The minikube logo

Meet kubectl

Before we start creating clusters, let us talk about kubectl. It is the official Kubernetes CLI and it interacts with your Kubernetes cluster's API server via its API. It is configured by default using the ~/.kube/config file, which is a YAML file that contains metadata, connection info, and authentication tokens or certificates for one or more clusters. Kubectl provides commands you can use to view your configuration and switch between clusters if it contains more than one. You can also point kubectl at a different config file by setting the KUBECONFIG environment variable. I prefer a third approach, which is keeping separate config file for each cluster and copying the active cluster's config file to ~/.kube/config (symlinks do not work).

We will discover together what kubectl can do along the way. The purpose here is just to avoid confusion when working with different clusters and configuration files.

Quick introduction to Minikube

Minikube is the most mature local Kubernetes cluster. It runs the latest stable Kubernetes release and it supports Windows, macOS, and Linux. It supports:

• LoadBalancer service type via Minikube tunnel
• NodePort service type via Minikube service
• Multiple clusters
• Filesystem mounts
• GPU support for machine learning
• RBAC
• Persistent volumes
• Ingress
• Dashboard via Minikube dashboard
• Custom container runtimes via the start --container-runtime flag
• Configuration API server and kubelet options via command-line flags
• Add-ons

Getting ready

There are some prerequisites to install before you can create the cluster itself. These include VirtualBox, the kubectl command-line interface to Kubernetes, and, of course, Minikube itself. Here is a list of the latest versions at the time of writing:

• VirtualBox: https://www.virtualbox.org/wiki/Downloads
• Kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/
• Minikube: https://kubernetes.io/docs/tasks/tools/install-minikube/

On Windows

Install VirtualBox and make sure kubectl and Minikube are on your path. I personally just throw all command-line programs I use into c:. You may prefer another approach. I use the excellent ConEMU to manage multiple consoles, terminals, and SSH sessions. It works with Command Prompt, PowerShell, PuTTY, Cygwin, msys, and Git-Bash. It does not get much better than that on Windows.

With Windows 10 Pro, you have the option to use the Hyper-V hypervisor. This is technically a better solution than VirtualBox, but it requires the Pro version of Windows and is completely Windows-specific. By using VirtualBox, these instructions are universal and will be easy to adapt to other versions of Windows, or other operating systems altogether. If you have Hyper-V enabled, you must disable it because VirtualBox cannot coexist with Hyper-V.

I recommend using PowerShell in administrator mode. You can add the following alias and function to your PowerShell profile:

Set-Alias -Name k -Value kubectl
function mk
{
  minikube-windows-amd64 `
  --show-libmachine-logs `
  --alsologtostderr      `
  @args
}

On macOS

On macOS, you have the option of using HyperKit instead of VirtualBox:

$ curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit \
  && chmod +x docker-machine-driver-hyperkit \
  && sudo mv docker-machine-driver-hyperkit /usr/local/bin/ \
  && sudo chown root:wheel /usr/local/bin/docker-machine-driver-hyperkit \
  && sudo chmod u+s /usr/local/bin/docker-machine-driver-hyperkit

You can add aliases to your .bashrc file (similar to the PowerShell alias and function on Windows):

alias k='kubectl'
alias mk='/usr/local/bin/minikube'

If you chose HyperKit instead of VirtualBox, you need to add the flag --vm-driver=hyperkit when starting the cluster.

It is also important to disable any VPN when using HyperKit.

Now, you can use k and mk and type less. The flags to Minikube in the mk function provide better logging and direct it to the console in addition to files (similar to tee).

Type mk version to verify Minikube is correctly installed and functioning:

$ mk version
minikube version: v1.10.1

Type k version to verify kubectl is correctly installed and functioning:

$ k version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
The connection to the server localhost:8080 was refused — did you specify the right host or port?
Unable to connect to the server: dial tcp 192.168.99.100:8443: getsockopt: operation timed out

Do not worry about the error on the last line. There is no cluster running, so kubectl cannot connect to anything. That is expected.

You can explore the available commands and flags for both Minikube and kubectl. I will not go over each command, only the commands I use.

Creating the cluster

The Minikube tool supports multiple versions of Kubernetes. At the time of writing, the latest version is 1.18.0, which is also the default:

$ mk start
  minikube v1.10.1 on darwin (amd64)
  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
  Configuring environment for Kubernetes v1.18.0 on Docker 19.03.8
  Pulling images ...
  Launching Kubernetes ...
  Verifying: apiserver proxy etcd scheduler controller dns
  Done! kubectl is now configured to use "minikube"

When you restart an existing stopped cluster, you will see the following output:

$ mk start
  minikube v1.10.1 on darwin (amd64)
  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
  Restarting existing virtualbox VM for "minikube" ...
  Waiting for SSH access ...
  Configuring environment for Kubernetes v1.18.0 on Docker 19.03.8
  Relaunching Kubernetes v1.18.0 using kubeadm ...
  Verifying: apiserver proxy etcd scheduler controller dns
  Done! kubectl is now configured to use "minikube"

Let us review what Minikube did behind the curtains for you. You will need to do a lot of it when creating a cluster from scratch:

1. Start a VirtualBox VM
2. Create certificates for the local machine and the VM
3. Download images
4. Set up networking between the local machine and the VM
5. Run the local Kubernetes cluster on the VM
6. Configure the cluster
7. Start all the Kubernetes control plane components
8. Configure kubectl to talk to the cluster

Troubleshooting

If something goes wrong during the process, try to follow the error messages. You can add the --alsologtostderr flag to get detailed error info to the console. Everything Minikube does is organized neatly under ~/.minikube. Here is the directory structure:

$ tree ~/.minikube -L 2 
/Users/gigi.sayfan/.minikube 
├── addons 
├── apiserver.crt 
├── apiserver.key 
├── ca.crt 
├── ca.key 
├── ca.pem 
├── cache 
│   ├── images 
│   ├── iso 
│   └── v1.15.0 
├── cert.pem 
├── certs 
│   ├── ca-key.pem 
│   ├── ca.pem 
│   ├── cert.pem 
│   └── key.pem 
├── client.crt 
├── client.key 
├── config 
├── files 
├── key.pem 
├── logs 
├── machines 
│   ├── minikube 
│   ├── server-key.pem 
│   └── server.pem 
├── profiles 
│   └── minikube 
├── proxy-client-ca.crt 
├── proxy-client-ca.key 
├── proxy-client.crt 
└── proxy-client.key
 13 directories, 19 files

Checking out the cluster

Now that we have a cluster up and running, let's peek inside.

First, let's ssh into the VM:

$ mk ssh
                         _             _             
            _         _ ( )           ( )            
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __   
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\ 
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/ 
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____) 
$ uname -a
Linux minikube 4.19.107 #1 SMP Mon May 11 14:51:04 PDT 2020 x86_64 GNU/Linux 
$

Great! That works. The weird marks symbols are ASCII art for "minikube." Now, let us start using kubectl, because it is the Swiss Army knife of Kubernetes and will be useful for all clusters (including federated clusters).

Disconnect from the VM via Ctrl + D or by typing:

$ logout

We will cover many of the kubectl commands throughout our journey. First, let us check the cluster status using cluster-info:

$ k cluster-info
Kubernetes master is running at https://192.168.99.103:8443
KubeDNS is running at https://192.168.99.103:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use kubectl cluster-info dump.

You can see that the master is running properly. To see a much more detailed view of all the objects in the cluster as JSON, type k cluster-info dump. The output can be a little daunting, so let us use more specific commands to explore the cluster.

Let us check out the nodes in the cluster using get nodes:

$ k get nodes
NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   28m   v1.16.3

So, we have one node called minikube. To get a lot more information about it, type k describe node minikube.

The output is verbose; I will let you try it yourself.

Doing work

We have a nice empty cluster up and running (well, not completely empty as the DNS service and dashboard run as pods in the kube-system namespace). It is time to deploy some pods:

$ k create deployment echo --image=gcr.io/google_containers/echoserver:1.8
deployment.apps/echo created

Let us check out the pod that was created:

$ k get pods
NAME                   READY   STATUS    RESTARTS   AGE
echo-855975f9c-r6kj8   1/1     Running   0          2m11s

To expose our pod as a service, type the following:

$ k expose deployment echo --type=NodePort --port=8080
service/echo exposed

Exposing the service as type NodePort means that it is exposed to the host on some port. But it is not the 8080 port we ran the pod on. Ports get mapped in the cluster. To access the service, we need the cluster IP and the exposed port:

$ mk ip
192.168.99.103
$ k get service echo --output="jsonpath='{.spec.ports[0].nodePort}'"
31800

Now, we can access the echo service, which returns a lot of information.

Replace the IP address and port with the results of the previous commands:

$ curl http://192.168.99.103:31800/hi
    Hostname: echo-855975f9c-r6kj8
    
   Pod Information:
        -no pod information available-
    
   Server values:
        server_version=nginx: 1.13.3 - lua: 10008
    
   Request Information:
        client_address=172.17.0.1
        method=GET
        real path=/hi
        query=
        request_version=1.1
        request_uri=http://192.168.99.103:8080/hi
    
   Request Headers:
        accept=*/*
        host=192.168.99.103:31800
        user-agent=curl/7.64.0
    
   Request Body:
        -no body in request-

Congratulations! You just created a local Kubernetes cluster, deployed a service, and exposed it to the world.

Examining the cluster with the dashboard

Kubernetes has a very nice web interface, which is deployed, of course, as a service in a pod. The dashboard is well designed and provides a high-level overview of your cluster. It also lets you drill down into individual resources, view logs, edit resource files, and more. It is the perfect weapon when you want to check out your cluster manually. To launch it, type:

$ mk dashboard
  Enabling dashboard ...
  Verifying dashboard health ...
  Launching proxy ...
  Verifying proxy health ...
  Opening http://127.0.0.1:56853/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...

Minikube will open a browser window with the dashboard UI. Note that, on Windows, Microsoft Edge cannot display the dashboard. I had to run it myself on a different browser.

Here is the workloads view, which displays deployments, replica sets, replication controllers, and pods:

Figure 2.2: Kubernetes dashboard UI

It can also display daemon sets, stateful sets, and jobs, but we do not have any in this cluster.

In this section, we created a local single-node Kubernetes cluster on Windows, explored it a little bit using kubectl, deployed a service, and played with the web UI. In the next section, we will move on to a multi-node cluster.

In this section, we will create a multi-node cluster using KinD. We will also repeat the deployment of the echo server we deployed on Minikube and observe the differences. Spoiler alert – everything will be faster and easier!

Quick introduction to KinD

KinD stands for Kubernetes in Docker. It is a tool for creating ephemeral clusters (no persistent storage). It was built primarily for running the Kubernetes conformance tests. It supports Kubernetes 1.11+. Under the cover, it uses kubeadm to bootstrap Docker containers as nodes in the cluster. KinD is a combination of a library and a CLI. You can use the library in your code for testing or other purposes. KinD can create highly available clusters with multiple master nodes. Finally, KinD is a CNCF conformant Kubernetes installer. It better be if it is used for the conformance tests of Kubernetes itself :-).

KinD is super fast to start, but it has some limitations too: no persistent storage and no support for alternative runtimes yet, only Docker.

Let's install KinD and get going.

Installing KinD

You must have Docker installed as KinD is literally running as a Docker container. If you have Go 1.11+ installed, you can install the KinD CLI via:

$ GO111MODULE="on" go get sigs.k8s.io/kind@v0.8.1

Otherwise, on macOS, type:

$ curl -Lo ./kind-darwin-amd64 https://github.com/kubernetes-sigs/kind/releases/download/v0.8.1/kind-darwin-amd64
$ chmod +x ./kind-darwin-amd64
$ mv ./kind-darwin-amd64 /usr/local/bin/kind

On Windows, type (in PowerShell):

c:\> curl.exe -Lo kind-windows-amd64.exe https://github.com/kubernetes-sigs/kind/releases/download/v0.8.1/kind-windows-amd64
c:\> Move-Item .\kind-windows-amd64.exe c:\windows\kind.exe

Creating the cluster with KinD

Creating a cluster is super easy:

$ kind create cluster
Creating cluster "kind" ...
  Ensuring node image (kindest/node:v1.16.3) 
  Preparing nodes 
  Creating kubeadm config 
  Starting control-plane 
Cluster creation complete. You can now use the cluster with:
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info

KinD suggests that you export KUBECONFIG, but as I mentioned earlier, I prefer to copy the config file to ~/.kube/config so I do not have to export again if I want to access the cluster from another terminal window:

$ cp $(kind get kubeconfig-path --name="kind") ~/.kube/config

Now, we can access the cluster using kubectl:

$ k cluster-info
Kubernetes master is running at https://localhost:58560
KubeDNS is running at https://localhost:58560/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump.'

However, this creates a single-node cluster:

$ k get nodes
NAME                 STATUS   ROLES    AGE   VERSION
kind-control-plane   Ready    master   11m   v1.16.3

Let us delete it and create a multi-node cluster:

$ kind delete cluster
Deleting cluster "kind" ...

To create a multi-node cluster, we need to provide a configuration file with the specification of our nodes. Here is a configuration file that will create a cluster with one control-plane node and two worker nodes:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
- role: control-plane
- role: worker
- role: worker

Let us save the configuration file as kind-multi-node-config.yaml and create the cluster:

$ kind create cluster --config kind-multi-node-config.yaml
Creating cluster "kind" ...
  Ensuring node image (kindest/node:v1.16.3) 
  Preparing nodes   
  Creating kubeadm config 
  Starting control-plane 
  Joining worker nodes 
Cluster creation complete. You can now use the cluster with:
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info

Yeah, it works! We have a local three-node cluster now:

$ k get nodes
NAME                  STATUS     ROLES    AGE   VERSION
kind-control-plane    Ready      master   12m   v1.16.3
kind-worker           NotReady   <none>   11m   v1.16.3
kind-worker2          NotReady   <none>   11m   v1.16.3

KinD is also kind enough (see what I did there) to let us create highly available (HA) clusters with multiple control plane nodes for redundancy. Let us give it a try and see what it looks like with two control-plane nodes and two worker nodes:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
- role: control-plane
- role: control-plane
- role: worker
- role: worker

Let us save the configuration file as kind-ha-multi-node-config.yaml, delete the current cluster, and create a new HA cluster:

$ kind delete cluster
Deleting cluster "kind" ...
$ kind create cluster --config kind-ha-multi-node-config.yaml
Creating cluster "kind" ...
  Ensuring node image (kindest/node:v1.16.3) 
  Preparing nodes      
  Starting the external load balancer 
  Creating kubeadm config 
  Starting control-plane 
  Joining more control-plane nodes 
  Joining worker nodes 
Cluster creation complete. You can now use the cluster with:
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info

Hmmm... there is something new here. Now, KinD creates an external load balancer, as well as join more control-plane nodes before joining the worker nodes. The load balancer is necessary to distribute requests across all the control-plane nodes.

Note that the external load balancer does not show as a node using kubectl:

$ k get nodes
NAME                  STATUS   ROLES    AGE     VERSION
kind-control-plane    Ready    master   8m31s   v1.16.3
kind-control-plane2   Ready    master   8m14s   v1.16.3
kind-worker           Ready    <none>   7m35s   v1.16.3
kind-worker2          Ready    <none>   7m35s   v1.16.3

However, KinD has its own get nodes command, where you can see the load balancer:

$ kind get nodes
kind-control-plane2
kind-worker
kind-control-plane
kind-worker2
kind-external-load-balancer

Doing work with KinD

Let us deploy our echo service on the KinD cluster. It starts the same:

$ k create deployment echo --image=gcr.io/google_containers/echoserver:1.8 deployment.apps/echo created
$ k expose deployment echo --type=NodePort --port=8080
service/echo exposed

Checking our services, we can see the echo service front and center:

$  k get svc echo
NAME   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
echo   NodePort   10.105.48.21   <none>        8080:31550/TCP   3m5s

However, there is no external IP to the service. With Minikube, we got the IP of the Minikube node itself via $(minikube ip), and we can use it in combination with the node port to access the service. That is not an option with KinD clusters. Let us see how to use a proxy to access the echo service.

Accessing Kubernetes services locally though a proxy

In this section, we will go into a lot of detail about networking, services, and how to expose them outside the cluster.

Here, I am just showing how to get it done and keeping you in suspense for now. First, we need to run the kubectl proxy command, which exposes the API server, pods, and services on localhost:

$ k proxy &
[1] 10653
Starting to serve on 127.0.0.1:8001

Then, we can access the echo service though a specially crafted proxy URL that includes the exposed port (8080) and NOT the node port.

I use Httpie here. You can use curl too. To install Httpie, follow the instructions here: https://httpie.org/doc#installation:

$ http http://localhost:8001/api/v1/namespaces/default/services/echo:8080/proxy/
HTTP/1.1 200 OK
Content-Length: 534
Content-Type: text/plain
Date: Thu, 28 May 2020 21:27:56 GMT
Server: echoserver
Hostname: echo-74545d499-wqkn9
Pod Information:
    -no pod information available-
Server values:
    server_version=nginx: 1.13.3 - lua: 10008
Request Information:
    client_address=10.40.0.0
    method=GET
    real path=/
    query=
    request_version=1.1
    request_uri=http://localhost:8080/
Request Headers:
    accept=*/*
    accept-encoding=gzip, deflate
    host=localhost:8001
    user-agent=HTTPie/0.9.9
    x-forwarded-for=127.0.0.1, 172.17.0.1
    x-forwarded-uri=/api/v1/namespaces/default/services/echo:8080/proxy/
Request Body:
    -no body in request-

We will deep dive into exactly what is going on in a future chapter (Chapter 12, Serverless Computing on Kubernetes). Let us check out my favorite local cluster solution: k3d.

In this section, we'll create a multi-node cluster using k3d from Rancher. We will not repeat the deployment of the echo server because it is identical to the KinD cluster, including accessing it though a proxy. Spoiler alert – it is even faster and more user-friendly than KinD!

Quick introduction to k3s and k3d

Rancher created k3s, which is a lightweight Kubernetes distribution. Rancher says that k3s is five less than k8s, if that makes any sense. The basic idea is to remove features and capabilities that most people don't need, such as:

• Non-default features
• Legacy features
• Alpha features
• In-tree storage drivers
• In-tree cloud providers

However, the big ticket item is that k3s removed Docker and uses containerd instead. You can still bring Docker back if you depend on it.

Another major change is that k3s stores its state in a SQLite DB instead of etcd.

For networking and DNS, k3s uses Flannel and CoreDNS.

k3s also added a simplified installer that takes care of SSL and certificate provisioning.

The end result is astonishing – a single binary (less than 40 MB) that needs only 512 MB of memory.

Unlike Minikube and KinD, k3s is actually designed for production. The primary use case is for edge computing, IoT, and CI systems. It is optimized for ARM devices.

OK. That's k3s, but what's k3d? k3d takes all the goodness that is k3s, packages it in Docker (similar to KinD), and adds a friendly CLI to manage it.

Installing k3d

Installing k3d is as simple as:

$ curl -s https://raw.githubusercontent.com/rancher/k3d/master/install.sh | bash

The usual disclaimer is in effect – make sure to read the installation script before downloading and piping it to bash.

Creating the cluster with k3d

Are you ready to be amazed? Creating a single-node cluster with k3d takes less than 2 seconds!

$ time k3d create --workers 1
2020/05/28 17:07:36 Created cluster network with ID f09fde83314b059d1a442ec1d01fcd62e522e5f1d838121528c5a1ae582e3cbf
2020/05/28 17:07:36 Creating cluster [k3s-default]
2020/05/28 17:07:36 Creating server using docker.io/rancher/k3s:v1.17.3-k3s1...
2020/05/28 17:07:36 Booting 1 workers for cluster k3s-default
2020/05/28 17:07:37 Created worker with ID 8a6bd47f7a5abfbac5c396c45f13db04c7e18749ff4d2e054e737fe7f7843010
2020/05/28 17:07:37 SUCCESS: created cluster [k3s-default]
2020/05/28 17:07:37 You can now use the cluster with:
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info
real    0m1.896s
user    0m0.009s
sys 0m0.011s

What about a multi-node cluster? We saw that KinD was much slower, especially when creating a HA cluster with multiple control-plane nodes and an external load balancer.

Let's delete the single-node cluster first:

$ k3d delete
2020/05/28 17:08:42 Removing cluster [k3s-default]
2020/05/28 17:08:42 ...Removing 1 workers
2020/05/28 17:08:43 ...Removing server
2020/05/28 17:08:45 SUCCESS: removed cluster [k3s-default]

Now, let's create a cluster with three worker nodes. That takes a little over 5 seconds:

$ time k3d create --workers 3
2020/05/28 17:09:16 Created cluster network with ID 5cd1e01434edb1facdab28e563b78b605af416e2ad062dc121400c3f8a5d166c
2020/05/28 17:09:16 Creating cluster [k3s-default]
2020/05/28 17:09:16 Creating server using docker.io/rancher/k3s:v1.17.3-k3s1...
2020/05/28 17:09:17 Booting 3 workers for cluster k3s-default
2020/05/28 17:09:19 Created worker with ID 4b442116f8df7debecc9d70cee8ae8fb8f16783c0a8f111268be531f71dd54fa
2020/05/28 17:09:20 Created worker with ID 369879f1a38d60935908705f56b34a95caf6a44970beeb509c0cfb2047cd503a
2020/05/28 17:09:20 Created worker with ID d531937996fd25490276e32150b69aa2356c90cfcd1b480ab77ec3d2be08a2f6
2020/05/28 17:09:20 SUCCESS: created cluster [k3s-default]
2020/05/28 17:09:20 You can now use the cluster with:
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info
real    0m5.164s
user    0m0.011s
sys 0m0.019s

Let's verify the cluster works as expected:

$ export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
$ kubectl cluster-info
Kubernetes master is running at https://localhost:6443
CoreDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Here are the nodes. Note that there is just one master called k3d-k3s-default-server:

$ k get nodes
NAME                       STATUS   ROLES    AGE   VERSION
k3d-k3s-default-server     Ready    <none>   14h   v1.17.3-k3s1
k3d-k3s-default-worker-0   Ready    <none>   14h   v1.17.3-k3s1
k3d-k3s-default-worker-1   Ready    <none>   14h   v1.17.3-k3s1
k3d-k3s-default-worker-2   Ready    <none>   14h   v1.17.3-k3s1

You can stop and start clusters, create multiple clusters, and list existing clusters using the k3d CLI. Here are all the commands. Feel free to explore them further:

$ k3d
NAME:
   k3d - Run k3s in Docker!
USAGE:
   k3d [global options] command [command options] [arguments...]
VERSION:
   v1.7.0
AUTHORS:
   Thorsten Klein iwilltry42@gmail.com
   Rishabh Gupta r.g.gupta@outlook.com
   Darren Shepherd
COMMANDS:
     check-tools, ct  Check if docker is running
     shell            Start a subshell for a cluster
     create, c        Create a single- or multi-node k3s cluster in docker containers
     delete, d, del   Delete cluster
     stop             Stop cluster
     start            Start a stopped cluster
     list, ls, l      List all clusters
     get-kubeconfig   Get kubeconfig location for cluster
     help, h          Shows a list of commands or help for one command
GLOBAL OPTIONS:
   --verbose      Enable verbose output
   --help, -h     show help
   --version, -v  print the version

You can repeat the steps for deploying, exposing, and accessing the echo service on your own. It works just like KinD.

OK. We created clusters using Minikube, KinD, and k3d. Let's compare them so that you can decide which one works for you.

Minikube is the official local Kubernetes release. It is part of Kubernetes; it's very mature and very full featured. That said, it requires a VM and is both slow to install and to start. It can also get into trouble with networking at arbitrary times and sometimes the only remedy is deleting the cluster and rebooting. Also, Minikube supports a single node only. I suggest using Minikube only if it supports some feature that you need that is not available in either KinD or k3d.

KinD is much faster than Minikube and is used for Kubernetes conformance tests, so by definition, it is a conformant Kubernetes distribution. It is the only local cluster solution that provides HA clusters with multiple control-plane nodes. It is also designed to be used as a library, which I don't find as a big attraction because it is very easy to automate CLIs from code. The main downside of KinD for local development is that it is ephemeral. I recommend using KinD if you contribute to Kubernetes itself and want to test against it.

k3d is the clear winner for me. It's lightning fast, supports multiple clusters, and supports multiple worker nodes per cluster. It's also easy to stop and start clusters without losing state.

Alright. Let's take a look at the cloud.