Parameter server architecture
In this section, we will dive into the system architecture of the parameter server paradigm. The domain knowledge requirements for this section are as follows:
- A Master/Worker architecture in distributed systems
- Client/Server communication
The parameter server architecture mainly consists of two roles: parameter server and worker. The parameter server can be regarded as the master node in the traditional Master/Worker architecture.
Workers are the computer nodes or GPUs that are responsible for model training. We split the total training data among all the workers. Each worker trains their local model with the training data partition that's been assigned to it.
The duties of parameter server are twofold:
- Aggregate model updates from all the workers.
- Update the model parameters held on the parameter server.
The following diagram depicts a simplified parameter server architecture with two workers and one...
