Chapter 8: Gradient Agreement as an Optimization Objective

1. When the gradients of all tasks are in the same direction, then it is called gradient agreement, and when the gradient of some tasks differ greatly from others, then it is called gradient disagreement. 
2. The update equation in gradient agreement can be expressed as   .
3. Weights are proportional to the inner product of the gradients of a task and the average of gradients of all of the tasks in the sampled batch of tasks. 

4. The weights are calculated as follows:

   

5. The normalization factor is proportional to the inner product of  and . 
6. If the gradient of a task is in the same direction as the average gradient of all tasks in a sampled batch of tasks, then we can increase its weights so that it'll contribute more when updating our model parameter. Similarly,...