Tracing transformer paths in GPT

From an emerging General Purpose Technology perspective, fine-tuning OpenAI models can be summed as productivity improvements and diffusion limitations:

Improvements

• Automated dataset control
• OpenAI provides a data-preparation tool that will accept the data, try to correct the errors, and explain the potential issues.
• Intuitive fine-tuning
• OpenAI models can be fine-tuned with a few instructions.
• Synchronized metrics
• OpenAI can be synchronized with Weights & Biases' Wandb, to produce information on the fine-tuning process, guaranteeing a productive level of traceability.
• A Generative model can be fine-tuned for a completion (generative) task and a classification (discriminative) task, as shown in Figure 8.1:

Figure 8.1: A generative model can be generative and discriminative. Source: Tracing_Transformer_Paths_in_GPT.ipynb in Appendix1, Terminology

Diffusion

ChatGPT reached mainstream users as ready-to-use assistants. The diffusion of fine...