Learning from top solutions

In this section, we gather aspects of the top solutions that could allow us to rise above the level of the baseline solution. Keep in mind that the leaderboards (both public and private) in this competition were quite tight; this was due to a combination of a couple of factors:

• Noisy data: it was easy to get to 0.89 accuracy by correctly identifying a large part of the train data, and then each new correct one allowed for a tiny move upward
• Limited size of the data

Pretraining

The first and most obvious remedy to the issue of limited data size was pretraining: using more data. Pretraining a deep learning model on more data can be beneficial because it can help the model learn better representations of the data, which can in turn improve the performance of the model on downstream tasks. When a deep learning model is trained on a large dataset, it can learn to extract useful features from the data that are relevant to the task...