Chapter 5, BERT Variants II – Based on Knowledge Distillation

1. Knowledge distillation is a model compression technique in which a small model is trained to reproduce the behavior of a large pre-trained model. It is also referred to as teacher-student learning, where the large pre-trained model is the teacher and the small model is the student.
2. The output of the teacher network is called a soft target, and the prediction made by the student network is called a soft prediction.
3. In knowledge distillation, we compute the cross-entropy loss between the soft target and soft prediction and train the student network through backpropagation by minimizing the cross-entropy loss. The cross-entropy loss between the soft-target and soft-prediction is also known as the distillation loss.
4. The pre-trained BERT model has a large number of parameters and also high inference time, which makes it harder to use them on edge devices such as mobile phones. To solve this issue, we use DistilBERT, which...