Skills to use Apache Airflow effectively
It isn’t sufficient to just know Airflow; there are other skills that are required and will impact your team’s effectiveness at providing value with Airflow. As you’re reading this book, think about your teams’ skills in the following areas to understand if you need additional support or capabilities to make the most out of your platform:
- Python: DAGs and plugins are written in Python. As such, in order for Airflow to be effectively adopted, you should be able to write and understand Python code.
- Application testing: Your company should have relatively mature testing processes and practices in place to ensure that the plugins you write and the DAGs you author can be reasonably expected to function as intended. The team responsible for the health of your Airflow instance(s) and plugins should be able to help teams prepare for upgrades, and the teams responsible for writing workflows should be able to run automated tests prior to production deployments.
- Domain expertise: Knowing your data domain is the most critical skill for your success in adopting Airflow. No matter how much technical skill you have, if you don’t understand the business domain you’re working with, you won’t be successful in the short or long run.
- Application monitoring/alerting: You should have relatively mature observing, monitoring, and alerting capabilities available to you in order to make the most out of Airflow. Some common activities are things such as monitoring for application activity (that is, health checks), sending alerts to on-call team members, and automated messages about status.
These skills identified are key to making the most out of Apache Airflow and should be reviewed periodically to see how you are progressing and how you can improve in different areas.
