Activation functions
In Chapter 1, Neural Network Foundations with TF, we saw a few activation functions including sigmoid, tanh, and ReLU. In the section below, we compute the derivative of these activation functions.
Derivative of the sigmoid
Remember that the sigmoid is defined as 
(see Figure 14.6):
Figure 14.6: Sigmoid activation function
The derivative can be computed as follows:
Therefore the derivative of 
can be computed as a very simple form: 
.
Derivative of tanh
Remember that the arctan function is defined as 
as seen in Figure 14.7:
Figure 14.7: Tanh activation function
If you remember that 
and 
, then the derivative is computed as:
Therefore the derivative of 
can be computed as a very simple form: 
.
Derivative of ReLU
The ReLU function is defined as 
(see Figure 14.8). The derivative of ReLU is:
Note that ReLU is non-differentiable at zero. However, it is differentiable anywhere else, and the...
