A2C revisited

We can design our A2C algorithm with many worker agents, just like the A3C algorithm. However, unlike A3C, A2C is a synchronous algorithm, meaning that in A2C, we can have multiple worker agents, each interacting with their own copies of the environment, and all the worker agents perform synchronous updates, unlike A3C, where the worker agents perform asynchronous updates.

That is, in A2C, each worker agent interacts with the environment, computes losses, and calculates gradients. However, it won't send those gradients to the global network independently. Instead, it waits for all other worker agents to finish their work and then updates the weights to the global network in a synchronous fashion. Performing synchronous weight updates reduces the inconsistency introduced by A3C.