Search icon Close icon
Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases now! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.

Change country

Modal Close icon
Country selected Country selected
country flag United States
Country selected
country flag United Kingdom
Country selected
country flag India
Country selected
country flag Germany
Country selected
country flag France
Country selected
country flag Canada
Country selected
country flag Russia
Country selected
country flag Spain
Country selected
country flag Brazil
Country selected
country flag Australia
Country selected
country flag Argentina
Country selected
country flag Austria
Country selected
country flag Belgium
Country selected
country flag Bulgaria
Country selected
country flag Chile
Country selected
country flag Colombia
Country selected
country flag Cyprus
Country selected
country flag Czechia
Country selected
country flag Denmark
Country selected
country flag Ecuador
Country selected
country flag Egypt
Country selected
country flag Estonia
Country selected
country flag Finland
Country selected
country flag Greece
Country selected
country flag Hungary
Country selected
country flag Indonesia
Country selected
country flag Ireland
Country selected
country flag Italy
Country selected
country flag Japan
Country selected
country flag Latvia
Country selected
country flag Lithuania
Country selected
country flag Luxembourg
Country selected
country flag Malaysia
Country selected
country flag Malta
Country selected
country flag Mexico
Country selected
country flag Netherlands
Country selected
country flag New Zealand
Country selected
country flag Norway
Country selected
country flag Philippines
Country selected
country flag Poland
Country selected
country flag Portugal
Country selected
country flag Romania
Country selected
country flag Singapore
Country selected
country flag Slovakia
Country selected
country flag Slovenia
Country selected
country flag South Africa
Country selected
country flag South Korea
Country selected
country flag Sweden
Country selected
country flag Switzerland
Country selected
country flag Taiwan
Country selected
country flag Thailand
Country selected
country flag Turkey
Country selected
country flag Ukraine
Country selected
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Getting Started with Google BERT

You're reading from   Getting Started with Google BERT Build and train state-of-the-art natural language processing models using BERT

Arrow left icon
Product type Paperback
Published in Jan 2021
Publisher Packt
ISBN-13 9781838821593
Length 352 pages
Edition 1st Edition
Languages
Tools
Concepts
Arrow right icon
Author (1):
Arrow left icon
Sudharsan Ravichandiran Sudharsan Ravichandiran
Author Profile Icon Sudharsan Ravichandiran
Sudharsan Ravichandiran
Sudharsan Ravichandiran is a data scientist and artificial intelligence enthusiast. He holds a Bachelors in Information Technology from Anna University. His area of research focuses on practical implementations of deep learning and reinforcement learning including natural language processing and computer vision. He is an open-source contributor and loves answering questions on Stack Overflow.
Read more
See other products by Sudharsan Ravichandiran
Arrow right icon
View More author details
Toc

Table of Contents (15) Chapters Close

Preface 1. Section 1 - Starting Off with BERT
2. A Primer on Transformers FREE CHAPTER 3. Understanding the BERT Model 4. Getting Hands-On with BERT 5. Section 2 - Exploring BERT Variants
6. BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT 7. BERT Variants II - Based on Knowledge Distillation 8. Section 3 - Applications of BERT
9. Exploring BERTSUM for Text Summarization 10. Applying BERT to Other Languages 11. Exploring Sentence and Domain-Specific BERT 12. Working with VideoBERT, BART, and More 13. Assessments 14. Other Books You May Enjoy

Questions

Let's put our knowledge to the test by answering the following questions:

  1. What is knowledge distillation?
  2. What are the soft target and soft prediction?
  3. Define distillation loss.
  4. What is the use of DistilBERT?
  5. What is the loss function of DistilBERT?
  6. How does transformer layer distillation work?
  7. How does prediction layer distillation work?
lock icon The rest of the chapter is locked
arrow left Previous Section
Section 7 of 8
Next Section arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at €18.99/month. Cancel anytime

Authors (1)

Profile icon Sudharsan Ravichandiran
Sudharsan Ravichandiran
Sudharsan Ravichandiran is a data scientist and artificial intelligence enthusiast. He holds a Bachelors in Information Technology from Anna University. His area of research focuses on practical implementations of deep learning and reinforcement learning including natural language processing and computer vision. He is an open-source contributor and loves answering questions on Stack Overflow.
Read more
See other products by Sudharsan Ravichandiran

Other recommended products

Related to this chapter
Left arrow icon
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Mastering Transformers
Mastering Transformers
Read more
Explore the accurate and fast fine-tuning capabilities of transformer-based language models and understand how they outperform traditional machine learning-based approaches when solving challenging NLU problems. Developers working with the Transformers architecture will be able to put their knowledge to work with this practical guide.
Read more
Sep 2021 12h 28m
Transformers for Natural Language Processing
Transformers for Natural Language Processing
Read more
Being the first book in the market to dive deep into the Transformers, it is a step-by-step guide for data and AI practitioners to help enhance the performance of language understanding and gain expertise with hands-on implementation of transformers using PyTorch, TensorFlow, Hugging Face, Trax, and AllenNLP.
Read more
Jan 2021 12h 48m
Transformers for Natural Language Processing
Transformers for Natural Language Processing
Read more
Being the first book in the market to dive deep into the Transformers, it is a step-by-step guide for data and AI practitioners to help enhance the performance of language understanding and gain expertise with hands-on implementation of transformers using PyTorch, TensorFlow, Hugging Face, Trax, and AllenNLP.
Read more
Jan 2021 12h 48m
Transformers for Natural Language Processing
Transformers for Natural Language Processing
Read more
Being the first book in the market to dive deep into the Transformers, it is a step-by-step guide for data and AI practitioners to help enhance the performance of language understanding and gain expertise with hands-on implementation of transformers using PyTorch, TensorFlow, Hugging Face, Trax, and AllenNLP.
Read more
Jan 2021 12h 48m
Right arrow icon