Summary

In this chapter, we learned how to detect shuffle in a process. We covered testing operations that cause a shuffle in Apache Spark. We also learned how to employ partitioning in the RDD. It is important to know how to use the API if partitioned data is needed, because RDD is still widely used, so we use the keyBy operations to reduce shuffle. We also learned how to use the custom partitioner to reduce shuffle.

In the next chapter, we'll learn how to save data in the correct format using the Spark API.