- See chapter_07_example_03.py.
-
A network that underfits is a network that hasn't reached its optimum, meaning it won't predict well with the evaluation data, because it fits poorly the training data (for now). It can be fixed by letting it train long enough, by adding more network capacity, and more data.
- A network that overfits is a network that has learned to predict the input but cannot generalize to values outside of its training set. It can be fixed by adding more data, by reducing the network capacity, or by using regularization techniques such as dropout.
- Early stopping.
- Read On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, which explains that a larger batch size leads to sharp minimizers, which in turn leads to poorer generalization. Therefore it is worse in terms of efficiency, but might...
