Summary
In this chapter, we discussed some of the options available at our disposal to tune AWS Glue Spark ETL jobs and AWS Glue crawlers based on the use case and understood how the procedure to tune a Glue ETL job or Glue crawler depends on data layout (input data type, partitioning structure, compression codec), crawler/job configuration, and downstream application/query engines. During our discussion on ETL job tuning, we explored different use cases and learned how to identify ETL jobs with straggler tasks and demanding stages and how we can optimize performance. We also discussed how to optimize ETL jobs with too many tasks and JDBC-/MongoDB-based ETL jobs to ensure we are using the resources allocated to the job to run quite efficiently.
We also outlined some common issues we may come across while working with an AWS Glue Spark ETL job and discussed different methods or steps to take to identify and mitigate such issues. It is important to note that while we discussed different...
