For this chapter, you should have Docker installed. Also, a computer with a minimum of 4 GB of RAM (8 GB is recommended) is required, as Docker can really consume a computer’s memory.

The code for this chapter is available on GitHub. Please refer to https://github.com/PacktPublishing/Bigdata-on-Kubernetes and access the Chapter01 folder.

Containers are an operating system-level virtualization method that we can use to run multiple isolated processes on a single host machine. Containers allow applications to run in an isolated environment with their own dependencies, libraries, and configuration files without the overhead of a full virtual machine (VM), which makes them lighter and more efficient.

If we compare containers to traditional VMs, they differ in a few ways. VMs virtualize at the hardware level, creating a full virtual operating system. Containers, on the other hand, virtualize at the operating system level. Because of that, containers share the host system’s kernel, whereas VMs each have their own kernel. This allows containers to have much faster startup times, typically in milliseconds compared to minutes for VMs (it is worth noting that in a Linux environment, Docker can leverage the capabilities of a Linux kernel directly. While running in a Windows system, however, it...

To get started with Docker, you can install it by using the package manager for your Linux distribution or install Docker Desktop for Mac/Windows machines.

Windows

To use Docker Desktop on Windows, you must turn on the WSL 2 feature. Refer to this link for detailed instructions: https://docs.microsoft.com/en-us/windows/wsl/install-win10.

After that, you can install Docker Desktop as follows:

1. Go to https://www.docker.com/products/docker-desktop and download the installer.
2. When the download is ready, double-click the installer and follow the prompts.

   You should ensure that the Use WSL 2 instead of Hyper-V option is selected on the Configuration page. This is the recommended usage. (If your system does not support WSL 2, this option will not be available. You can still run Docker with Hyper-V, though.)

3. After the installation is finished, close to complete and start Docker Desktop.

If you have any doubts, refer to the official documentation...

The very first Docker image we can run is the hello-world image. It is often used to test whether Docker is correctly installed and running.

hello-world

After the installation, open the terminal (Command Prompt in Windows) and run the following:

$ docker run hello-world

This command will pull the hello-world image from the Docker Hub public repository and run the application in it. If you can run it successfully, you will see this output:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
70f5ac315c5a: Pull complete
Digest: sha256:88ec0acaa3ec199d3b7eaf73588f4518c25 f9d34f58ce9a0df68429c5af48e8d
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the &quot...

Now, we will customize our own container images for running a simple data processing job and an API service.

Batch processing job

Here is a simple Python code for a batch processing job:

run.py

import pandas as pd
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
df = pd.read_csv(url, header=None)
df["newcolumn"] = df[5].apply(lambda x: x*2)
print(df.columns)
print(df.head())
print(df.shape)

This Python code loads a CSV dataset from a URL into a pandas DataFrame, adding a new column by multiplying an existing column by 2 and then printing out some information about the DataFrame (column names, first five rows, and size of the DataFrame). Type this code using your favorite code editor and save the file with the name run.py.

Normally, we test our code locally (whenever possible) to be sure it is working. To do that, first, you need to install the pandas library:

pip3 install...

In this chapter, we covered the fundamentals of containers and how to build and run them using Docker. Containers provide a lightweight and portable way to package applications and their dependencies so they can run reliably across environments.

You learned about key concepts such as images, containers, Dockerfiles, and registries. We installed Docker and ran simple containers such as NGINX and Julia to get hands-on experience. You built your own containers for a batch processing job and API service, defining Dockerfiles to package dependencies.

These skills allow you to develop applications and containerize them for smooth deployment anywhere. Containers are super useful as they ensure your software runs exactly as intended every time.

In the next chapter, we will look at orchestrating containers using Kubernetes to easily scale, monitor, and manage containerized applications. We will take a look at the most important Kubernetes concepts and components and learn how...