Collective communication

Besides the popular All-Reduce function, collective communication has a wide family of message-passing functions. We will discuss several of the most important collective communication functions in this section, namely Broadcast, Gather, and All-Gather.

Broadcast

Broadcast is also widely used in the All-Reduce architecture. For example, we can use Broadcast to distribute the initial model weights among all the workers. The following diagram shows an example of the Broadcast function in a three-workers setting:

Figure 2.13 – Broadcast in a three-workers setting

As we can see, initially, Worker 1 holds a value of a. Worker 2 holds a value of b, while Worker 3 holds a value of c. Here, we conduct a broadcast from Worker 1 to all the other workers.

After this broadcast operation, Worker 1 still holds a value of a. However, Worker 2 now holds values a and b. On the other hand, Worker 3 holds values a and c.

For this...