How one token sparked an AI revolution
One token produced an AI revolution and has opened the door to AI in every domain and application.
ChatGPT with GPT-4, PaLM 2, and other LLMs have a unique way of producing text.
In LLMs, a token is a minimal word part. The token is where an LLM starts and ends.
For example, the word including could become includ + ing, representing two tokens. GPT models predict tokens based on the hundreds of billions of tokens in its training dataset. Examine the graph in Figure A.7 of an OpenAI GPT model that is making an interference to produce a token:
Figure A.7: GPT inference graph built in Python with NetworkX
It may come as a surprise, but the only parts of this figure controlled by the model are “Model” and “Output Generation,” which produce raw logits. All the rest is in the pipeline.
To understand the pipeline, we will first go through a description of these steps:
- Tokenization:...
