Appendix IV — Custom Text Completion with GPT-2

This appendix, relating to Chapter 7, The Rise of Suprahuman Transformers with GPT-3 Engines, describes how to customize text completion with a GPT-2 model.

This appendix shows how to build a GPT-2 model, train it, and interact with custom text in 12 steps.

Open Training_OpenAI_GPT_2.ipynb, which is in the GitHub repository of this appendix. You will notice that the notebook is also divided into the same 12 steps and cells as this appendix.

Run the notebook cell by cell. The process is tedious, but the result produced by the cloned OpenAI GPT-2 repository is gratifying. We are not using the GPT-3 API or a Hugging Face wrapper.

We are getting our hands dirty to see how the model is built and trained. You will see some deprecation messages, but we need to get inside the model, not the wrappers or the API. However, the effort is worthwhile.

Let’s begin by activating the GPU.