Orchestrating ML workflows using Cloud Composer (managed Airflow)
Cloud Composer is a workflow orchestration service on Google Cloud that is built upon the open source project of Apache Airflow. The key difference is that Composer is fully managed and also integrates with other GCP tooling very easily. With Cloud Composer, we can write, execute, schedule, or monitor our workflows that are also supported across multi-cloud and hybrid environments. Composer pipelines are DAGs that can be easily defined and configured using Python. It comes with a rich library of connectors that let us deploy our workflows instantly with one click. Graphical representations of workflows on the Google Cloud console make monitoring and troubleshooting quite convenient. Automatic synchronization of our DAGs ensures that our jobs always stay on schedule.
Cloud Composer is commonly used by data scientists and data engineers to build complex data pipelines (ETL or ELT pipelines). It can also be used as an...