Designing a universal text-to-text model
Google’s NLP technical revolution started with Vaswani et al. (2017), the Original Transformer, in 2017. Attention Is All You Need toppled 30+ years of artificial intelligence belief in RNNs and CNNs applied to NLP tasks. It took us from the Stone Age of NLP/NLU to the 21st century in a long-overdue evolution.
Chapter 7, The Generative AI Revolution with ChatGPT, summed up a second revolution that boiled up and erupted between Google’s Vaswani et al. (2017) Original Transformer, OpenAI’s Brown et al. (2020) GPT-3 transformers, and now ChatGPT ’s, GPT-4 models. The Original Transformer was focused on performance to prove that attention was all we needed for NLP/NLU tasks.
OpenAI’s second revolution, through GPT-3, focused on taking transformer models from fine-tuned pretrained models to few-shot trained models that required no fine-tuning. ChatGPT with GPT-4 continued the progression that will continue...
