Hidden units and architecture design

In the next section, we'll recap artificial neural networks; they can do a good job in classification tasks such as classifying handwritten digits.

Suppose we have the network shown in Figure 1:

Figure 1: Simple FNN with one hidden layer

As mentioned earlier, the leftmost layer in this network is called the input layer, and the neurons within the layer are called input neurons. The rightmost or output layer contains the output neurons, or, as in this case, a single output neuron. The middle layer is called a hidden layer, since the neurons in this layer are neither inputs nor outputs. The term hidden perhaps sounds a little mysterious—the first time I heard the term, I thought it must have some deep philosophical or mathematical significance—but it really means not an input and not an output. It means nothing else. The preceding...