Preface

Transformers are a game-changer for Natural Language Understanding (NLU), a subset of Natural Language Processing (NLP), which has become one of the pillars of artificial intelligence in a global digital economy.

Transformer models mark the beginning of a new era in artificial intelligence. Language understanding has become the pillar of language modeling, chatbots, personal assistants, question answering, text summarizing, speech-to-text, sentiment analysis, machine translation, and more. We are witnessing the expansion of social networks versus physical encounters, e-commerce versus physical shopping, digital newspapers, streaming versus physical theaters, remote doctor consultations versus physical visits, remote work instead of on-site tasks, and similar trends in hundreds of more domains. It would be incredibly difficult for society to use web browsers, streaming services, and any digital activity involving language without AI language understanding. The paradigm shift of our societies from physical to massive digital information forced artificial intelligence into a new era. Artificial intelligence has evolved to billion-parameter models to face the challenge of trillion-word datasets.

The Transformer architecture is both revolutionary and disruptive. It breaks with the past, leaving the dominance of RNNs and CNNs behind. BERT and GPT models abandoned recurrent network layers and replaced them with self-attention. Transformer models outperform RNNs and CNNs. The 2020s are experiencing a major change in AI.

Transformer encoders and decoders contain attention heads that train separately, parallelizing cutting-edge hardware. Attention heads can run on separate GPUs opening the door to billion-parameter models and soon-to-come trillion-parameter models. OpenAI trained a 175 billion parameter GPT-3 Transformer model on a supercomputer with 10,000 GPUs and 285,000 CPU cores.

The increasing amount of data requires training AI models at scale. As such, transformers pave the way to a new era of parameter-driven AI. Learning to understand how hundreds of millions of words fit together in sentences requires a tremendous amount of parameters.

Transformer models such as Google BERT, OpenAI’s GPT-3, ChatGPT, and GPT-4 have taken emergence to another level. Transformers can perform hundreds of NLP tasks they were not trained for.

Transformers can also learn image classification and reconstruction by embedding images as sequences of words. This book will introduce you to cutting-edge computer vision transformers such as Vision Transformers (ViT), CLIP, and DALL-E.

Foundation models are fully trained transformer models that can carry out hundreds of tasks without fine-tuning. Foundation models at this scale offer the tools we need in this massive information era. 

Think of how many humans it would take to control the content of the billions of messages posted on social networks per day to decide if they are legal and ethical before extracting the information they contain.

Think of how many humans would be required to translate the millions of pages published each day on the web. Or imagine how many people it would take to manually control the millions of messages made per minute!

Finally, think of how many humans it would take to write the transcripts of all of the vast amount of hours of streaming published per day on the web. Finally, think about the human resources required to replace AI image captioning for the billions of images that continuously appear online.

This book will take you from developing code to prompt design, a new “programming” skill that controls the behavior of a transformer model. Each chapter will take you through the key aspects of language understanding from scratch in Python, PyTorch, and TensorFlow.

You will learn the architecture of the original Transformer, Google BERT, OpenAI GPT-3, T5, and several other models. The last chapter builds on the knowledge acquired from the previous sixteen chapters, showing you the enhanced capabilities of ChatGPT and GPT-4. You will fine-tune transformers, train models from scratch, and learn to use powerful APIs. Facebook, Google, Microsoft, and other big tech corporations share large datasets for us to explore.

You will keep close to the market and its demand for language understanding in many fields such as media, social media, and research papers, for example. Among hundreds of AI tasks, we need to summarize the vast amounts of data for research, translate documents for every area of our economy, and scan all social media posts for ethical and legal reasons.

Throughout the book, you will work hands-on with Python, PyTorch, and TensorFlow. You will be introduced to the key AI language understanding neural network models. You will then learn how to explore and implement transformers.

You will learn the new skills required to become an Industry 4.0 AI Specialist in this disruptive AI era. The book aims to give readers the knowledge and tools for Python deep learning needed for effectively developing the key aspects of language understanding.