Implementing the Activation Function Layer
Now, we will apply the idea of a computational graph to a neural network. Here, we will implement the "layers" that constitute a neural network in one class using the ReLU and Sigmoid layers, which are activation functions.
ReLU Layer
A Rectified Linear Unit (ReLU) is used as an activation function and is expressed by the following equation (5.7):
|
|
(5.7) |
From the preceding equation (5.7), you can obtain the derivative of y with respect to x with equation (5.8):
|
|
(5.8) |
As equation (5.8) shows, if the input in forward propagation, x, is larger than 0, backward propagation passes the upstream value downstream without changing it. Meanwhile, if x is 0 or smaller in forward propagation, the signal stops there in backward propagation. You...
