Chapter 7: Implementing Model Parallel Training and Serving Workflows
In this chapter, we will discuss how to implement a simple model parallelism pipeline. As opposed to data parallelism, where each GPU holds a full copy of a model, in model parallelism, we need to split a model properly among all GPUs in use.
Before diving into the details, we'll qualify our discussion with the following assumptions about both hardware and workload:
- We will use homogenous GPUs for the same model parallel training or serving job.
- Each model training or serving task will occupy the whole hardware exclusively, which means there will be no preemption or interruption during the running of our model training or serving task.
- For GPUs within a machine, they are connected with either PCIe, NVLink, or NVSwitch.
- For GPUs among different machines, they are connected with general Ethernet links of 10 Gbps to 100 Gbps bandwidth.
- For the model parallel training part, we will mainly...
