Hyperparameter tuning – finding the sweet spot

Tuning hyperparameters is an important step in optimizing the performance of ML models, including LLMs. Let’s look at a systematic approach to hyperparameter tuning:

• Understand the hyperparameters: Begin by understanding the hyperparameters that influence model performance. In LLMs, these can include learning rate, batch size, number of layers, number of attention heads, dropout rate, and activation functions, among others. The choice of values for these hyperparameters can affect the balance between memory requirements and training efficiency.
• Establish a baseline: Start with a set of default hyperparameters to establish a baseline performance. This can either come from the literature, default settings in popular frameworks, or empirical guesses.
• Manual tuning: Initially, perform some manual tuning based on intuition and experience to see how different hyperparameters affect performance. This can help set...