Building neural networks

This practical section will start with implementing a shallow network from scratch, followed by a deep network with two layers using scikit-learn. We will then implement a deep network with TensorFlow and PyTorch.

Implementing neural networks from scratch

To demonstrate how activation functions work, we will use sigmoid as the activation function in this example.

We first define the sigmoid function and its derivative function:

>>> def sigmoid_derivative(z):
...     return sigmoid(z) * (1.0 - sigmoid(z))

You can derive the derivative yourself if you are interested in verifying it.

We then define the training function, which takes in the training dataset, the number of units in the hidden layer (we will only use one hidden layer as an example), and the number of iterations:

>>> def train(X, y, n_hidden, learning_rate, n_iter):
...     m, n_input = X.shape
...     W1 = np.random.randn(n_input, n_hidden)
...    ...