From task-agnostic models to multimodal vision transformers
Foundation Models, as we saw in Chapter 1, What Are Transformers?, have two distinct and unique properties:
- Emergence: Transformer models that qualify as Foundation Models can perform tasks they were not trained for. They are large models trained on supercomputers. They are not trained to learn specific tasks like many other models. Foundation Models learn how to understand sequences.
- Homogenization: The same model can be used across many domains with the same fundamental architecture. Foundation Models can learn new skills through data faster and better than any other model.
OpenAI ChatGPT models (GPT 3 and GPT 4), Google PaLM, and Google BERT (only the BERT models trained by Google) are task-agnostic Foundation Models. These task-agnostic models lead directly to ViT, CLIP, and DALL-E models.
The level of abstraction of transformer models leads to multimodal neurons. Multimodal neurons can process...
