Optimization with Adam

SGD, in contrast to batch gradient descent, performs a parameter update for each training example, x⁽ⁱ⁾ and label y⁽ⁱ⁾:

Θ = Θ - η∇_Θj(Θ, x⁽ⁱ⁾, y⁽ⁱ⁾)

Adaptive Moment Estimation (Adam) computes adaptive learning rates for each parameter. Like AdaDelta, Adam not only stores the decaying average of past squared gradients but additionally stores the momentum change for each parameter. Adam works well in practice and is one of the most used optimization methods today.

Adam stores the exponentially decaying average of past gradients (mt) in addition to the decaying average of past squared gradients (like Adadelta and RMSprop). Adam behaves like a heavy ball with friction running down the slope leading to a flat minima in the error surface. Decaying averages of past and past squared gradients mt and vt are computed with the following formulas:

m_t=β₁m_t−1+(1−β₁)g_t

v_t=β₂v_t−1+(1−β₂)g_t

m_t and v_t are estimates of the first moment (the mean) and the second moment (the uncentered variance...