Deploying Airflow on Kubernetes

Airflow deployment on Kubernetes is very straightforward. Nevertheless, there are some important details in the Helm chart configuration that we need to pay attention to.

First, we will download the most recent Helm chart to our local environment:

helm repo add apache-airflow https://airflow.apache.org

Next, we need to configure a custom_values.yaml file to change the default deployment configurations for the chart. An example of this YAML file is available at https://github.com/PacktPublishing/Bigdata-on-Kubernetes/blob/main/Chapter08/airflow/custom_values.yaml. We will not go through the entire file but just the most important configurations that are needed for this deployment:

1. In the defaultAirflowTag and airflowVersion parameters, make sure to set 2.8.3. This is the latest Airflow version available for the 1.13.1 Helm chart version.
2. The executor parameter should be set to KubernetesExecutor. This guarantees that Airflow will use...