Installing and configuring Apache Airflow

Apache Airflow performs the same role as Apache NiFi; however, it allows you to create your data flows using pure Python. If you are a strong Python developer, this is probably an ideal tool for you. It is currently one of the most popular open source data pipeline tools. What it lacks in a polished GUI – compared to NiFi – it more than makes up for in the power and freedom to create tasks.

Installing Apache Airflow can be accomplished using pip. But, before installing Apache Airflow, you can change the location of the Airflow install by exporting AIRFLOW_HOME. If you want Airflow to install to opt/airflow, export the AIRLFOW_HOME variable, as shown:

export AIRFLOW_HOME=/opt/airflow

The default location for Airflow is ~/airflow, and for this book, this is the location I will use. The next consideration before installing Airflow is to determine which sub-packages you want to install. If you do not specify any, Airflow...