You're reading from Mastering PyTorch Build powerful neural network architectures using advanced PyTorch 1.x features

Product type Paperback

Published in Feb 2021

Publisher Packt

ISBN-13 9781789614381

Length 450 pages

Edition 1st Edition

Languages

Python

Tools

PyTorch

Concepts

Deep Learning

Author (1):

Ashish Ranjan Jha

View More author details

Table of Contents (20) Chapters

Preface

1. Section 1: PyTorch Overview

2. Chapter 1: Overview of Deep Learning using PyTorch FREE CHAPTER

3. Chapter 2: Combining CNNs and LSTMs

4. Section 2: Working with Advanced Neural Network Architectures

5. Chapter 3: Deep CNN Architectures

6. Chapter 4: Deep Recurrent Model Architectures

7. Chapter 5: Hybrid Advanced Models

8. Section 3: Generative Models and Deep Reinforcement Learning

9. Chapter 6: Music and Text Generation with PyTorch

10. Chapter 7: Neural Style Transfer

11. Chapter 8: Deep Convolutional GANs

12. Chapter 9: Deep Reinforcement Learning

13. Section 4: PyTorch in Production Systems

14. Chapter 10: Operationalizing PyTorch Models into Production

15. Chapter 11: Distributed Training

16. Chapter 12: PyTorch and AutoML

17. Chapter 13: PyTorch and Explainable AI

18. Chapter 14: Rapid Prototyping with PyTorch

19. Other Books You May Enjoy

Leave a review - let other readers know what you think

Building a transformer model for language modeling

In this section, we will explore what transformers are and build one using PyTorch for the task of language modeling. We will also learn how to use some of its successors, such as BERT and GPT, via PyTorch's pretrained model repository. Before we start building a transformer model, let's quickly recap what language modeling is.

Reviewing language modeling

Language modeling is the task of figuring out the probability of the occurrence of a word or a sequence of words that should follow a given sequence of words. For example, if we are given French is a beautiful _____ as our sequence of words, what is the probability that the next word will be language or word, and so on? These probabilities are computed by modeling the language using various probabilistic and statistical techniques. The idea is to observe a text corpus and learn the grammar by learning which words occur together and which words never occur together...