Configuring sensors
Under the operator’s umbrella, we have sensors. Sensors are designed to wait to execute a task until something happens. For example, a sensor triggers a pipeline (or task) when a file lands in an HDFS folder, as shown here: https://airflow.apache.org/docs/apache-airflow-providers-apache-hdfs/stable/_api/airflow/providers/apache/hdfs/sensors/hdfs/index.html. As you might be wondering, there are also sensors for specific schedules or time deltas.
Sensors are a fundamental part of creating an automated and event-driven pipeline. In this recipe, we will configure a weekday sensor, which executes our data pipeline on a specific day of the week.
Getting ready
Refer to the Getting ready section in the Configuring Airflow recipe for this recipe since we will handle it with the same technology.
Besides that, let’s put a JSON file to the following path inside the Airflow folder: files_to_test/sensors_files/.
In my case, I will use the github_events...
