You're reading from Python Deep Learning - Third Edition

Product type Book

Published in Nov 2023

Publisher Packt

ISBN-13 9781837638505

Pages 362 pages

Edition 3rd Edition

Languages

Concepts

GPT/LLMs

Author (1):

Ivan Vasilev

Table of Contents (17) Chapters

Preface

1. Part 1:Introduction to Neural Networks

2. Chapter 1: Machine Learning – an Introduction

3. Chapter 2: Neural Networks

4. Chapter 3: Deep Learning Fundamentals

5. Part 2: Deep Neural Networks for Computer Vision

6. Chapter 4: Computer Vision with Convolutional Networks

7. Chapter 5: Advanced Computer Vision Applications

8. Part 3: Natural Language Processing and Transformers

9. Chapter 6: Natural Language Processing and Recurrent Neural Networks

10. Chapter 7: The Attention Mechanism and Transformers

11. Chapter 8: Exploring Large Language Models in Depth

12. Chapter 9: Advanced Applications of Large Language Models

13. Part 4: Developing and Deploying Deep Neural Networks

14. Chapter 10: Machine Learning Operations (MLOps)

15. Index

Why subscribe?

16. Other Books You May Enjoy

Understanding the attention mechanism

In this section, we’ll discuss several iterations of the attention mechanism in the order that they were introduced.

Bahdanau attention

The first attention iteration (Neural Machine Translation by Jointly Learning to Align and Translate, https://arxiv.org/abs/1409.0473), known as Bahdanau attention, extends the seq2seq model with the ability for the decoder to work with all encoder hidden states, not just the last one. It is an addition to the existing seq2seq model, rather than an independent entity. The following diagram shows how Bahdanau attention works:

Figure 7.2 – The attention mechanism

Don’t worry—it looks scarier than it is. We’ll go through this diagram from top to bottom: the attention mechanism works by plugging an additional context vector, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:m="http://schemas.openxmlformats.org/officeDocument/2006/math"><mml:msub><mml:mrow><mml:mi mathvariant="bold">c</mml:mi></mml:mrow><mml:mrow><mml:mi>t</mml:mi></mml:mrow></mml:msub></mml:math> , between the encoder and the decoder. The hidden decoder state <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:m="http://schemas.openxmlformats.org/officeDocument/2006/math"><mml:msub><mml:mrow><mml:mi mathvariant="bold">s</mml:mi></mml:mrow><mml:mrow><mml:mi>t</mml:mi></mml:mrow></mml:msub></mml:math> at time t is now a function not only of the hidden state and decoder output at...