Introducing Amazon MWAA
Managing big data platforms isn't typically part of the ML practitioner's portfolio of tasks. Oftentimes, the ML practitioner and data engineering teams rely on the infrastructure and operations teams to manage these platforms.
Including Airflow as part of the big data infrastructure means that these platform teams must now manage additional compute resources, orchestrate their deployment, update software and operating system patches, and monitor these resources to ensure that they are constantly addressing workload scaling requirements and other Service-Level Agreements (SLAs).
AWS offers multiple big data managed services, such as EMR and Glue, to help offload these management tasks from the platform teams and, in November 2020, AWS launched Amazon MWAA to help offload the management of the Airflow platform. With MWAA, the platform teams can run a highly available and scalable Airflow cluster without having to individually provision, update...