Tricks in training

In this section, we will talk about a few techniques that can help to train a better network, including how to initialize weights, tips for optimization parameters, and how to reduce overfitting.

Weight initialization

Following techniques are involved in weight initialization:

• All-zero
• Random initialization
• ReLU initialization
• Xavier initialization

All-zero

First, do NOT use all-zero initialization. Given proper data normalization, it is expected that roughly half of the network weights will be positive and half will be negative. However, this does...