Section 3 – Advanced Parallelism Paradigms

In this section, we will learn state-of-the-art techniques on top of traditional data and model parallelism approaches. First, we will understand the concept of hybrid data-model parallelism. Second, we will discuss federated learning and edge device learning. Third, we will discuss elastic and in-parallel model training/inference in multitenant clusters or cloud environments. Finally, we will look at some more advanced techniques for further accelerating in-parallel model training and serving.

This section comprises the following chapters:

• Chapter 9, A Hybrid of Data and Model Parallelism
• Chapter 10, Federated Learning and Edge Devices
• Chapter 11, Elastic Model Training and Serving
• Chapter 12, Advanced Techniques for Further Speed-Ups