Queuing Tasks with Redis

Receiving requests, performing an action, and then returning a response to the user can solve a lot of problems in web programming. However, there are times when this simple approach will simply not cut it. For instance, when I was working at MonolithAi, we had a functionality where the user would be able to put in data and parameters and then train a machine learning model on that data at a click of a button. However, trying to train a machine learning model before sending a response to the user would simply take too long. The connection would probably time out. To solve this, we had a Redis queue and a pool of workers consuming tasks. The training task would be put into the queue and one of the workers would work on training the model when they got round to it. The HTTP server would accept the request from the user, post the training task to the queue, and respond to the user that the task was posted. When the model was trained, the user would get an update...