Logging and Monitoring Your Data Ingest in Airflow
We already know how vital logging and monitoring are to manage applications and systems, and Airflow is no different. In fact, Apache Airflow already has built-in modules to create logs and export them. But what about improving them?
In the previous chapter, Putting Everything Together with Airflow, we covered the fundamental aspects of Airflow, how to start our data ingestion, and how to orchestrate a pipeline and use the best data development practices. Now, let’s put into practice the best techniques to enhance logging and monitor Airflow pipelines.
In this chapter, you will learn the following recipes:
- Creating basic logs in Airflow
- Storing log files in a remote location
- Configuring logs in
airflow.cfg - Designing advanced monitoring
- Using notification operators
- Using SQL operators for data quality
