The activation functions
As you can see from the preceding section, the activation function is part of the training process. The purpose of the activation function is to transform the weighted-sum input to the nodes: non-linearize and change the output range. There are many activation functions in neural networks. We will discuss some of the most used ones: the sigmoid function, the tanh activation function, the ReLu function, and the LeakyReLU function. Figure 5.4 shows the curves of these functions:
Figure 5.4 – Activation functions
Let’s inspect each of the preceding activation functions as follows:
- The sigmoid activation function was discussed earlier ithe The cost function section. We use the sigmoid function to change continuous values to a range between 0 and 1, which fits the models to predict the probability as an output.
- The tanh activation function is very similar to sigmoid, but the output is from -1 to +1 and thus it...
