Airflow for orchestration

Orchestration is the process of automating the workflow/pipeline that is to manage the task of scheduling the tasks, making coordination between tasks, and managing the created workflow. There are multiple tools available for automating the workflow such as Oozie, Azkaban, Jenkins, and so on. 

We have observed that people don't spend much time on workflow orchestration and the impact of scheduling failure or rerun. This causes big problems in later stages and then it will be difficult to manage the kind of problem it creates. In this section, we will learn about Airflow, which is the new generation orchestration tool for Hadoop applications. 

The user interface of Airflow is simple and easy to manage, and gives user's the flexibility to use and manage the workflows. The pipelines in Airflow are represented by DAG (Direct Acyclic Graph...