Chapter 9: A Hybrid of Data and Model Parallelism

In general, we have two different parallelism schemes—data parallelism and model parallelism. Each of them has advantages and disadvantages. In this chapter, we will try to leverage both data parallelism and model parallelism together. We call it a hybrid of data parallelism and model parallelism.

Before we discuss this further, we want to list our assumptions, as follows:

• Even though the advanced graphics processing unit (GPU) from NVIDIA now supports multi-tenancy, we still assume that one job occupies the whole GPU.
• When our training or serving job starts running, we do not allow job preemption or system interruption.
• We assume the use of homogenous GPUs for a single job.
• We assume the interconnects between GPUs within a machine are NVLink or NVSwitch.
• We assume the links among different machines use traditional Ethernet or InfiniBand (IB).
• We assume intra-machine communication bandwidth is...