A note on TensorFlow and automatic differentiation
TensorFlow can automatically calculate derivatives, a feature called Automatic Differentiation. This is achieved by using the chain rule. Every node in the computational graph (see Chapter 2, TensorFlow 1.x and 2.x) has an attached gradient operation for calculating the derivatives of input with respect to output. After that, the gradients with respect to parameters are automatically computed during backpropagation.
Automatic differentiation is a very important feature because you do not need to handcode new variations of backpropagation for each new model of a neural network. This allows quick iteration and running many experiments faster.