Airflow integration with other tools

We will take the DAG code developed in the last section and rebuild it with some different tasks. Our DAG is going to download the Titanic data, write it to a PostgreSQL table, and write it as a CSV file to Amazon S3. Also, we will create a view with a simple analysis on Postgres directly from Airflow:

1. Create a new Python file in the dags folder and name it postgres_aws_dag.py. The first part of our code will define the necessary modules. Note that, this time, we are importing the PostgresOperator class to interact with this database and the Variable class. This will help us manage secrets and parameters in Airflow. We are also creating an SQLAlchemy engine to connect to a local Postgres database and creating an S3 client that will allow writing files to S3:

   from airflow.decorators import task, dag
   from airflow.models import Variable
   from airflow.providers.postgres.operators.postgres import PostgresOperator
   from datetime import datetime
   import...