Chapter 3, Pretraining a RoBERTa Model from Scratch

1. RoBERTa uses a byte-level byte-pair encoding tokenizer. (True/False)

   True.

2. A trained Hugging Face tokenizer produces merges.txt and vocab.json. (True/False)

   True.

3. RoBERTa does not use token type IDs. (True/False)

   True.

4. DistilBERT has 6 layers and 12 heads. (True/False)

   True.

5. A transformer model with 80 million parameters is enormous. (True/False)

   False. 80 million parameters is a small model.

6. We cannot train a tokenizer. (True/False)

   False. A tokenizer can be trained.

7. A BERT-like model has 6 decoder layers. (True/False)

   False. BERT contains 6 encoder layers, not decoder layers.

8. Masked language modeling predicts a word contained in a mask token in a sentence. (True/False)

   True.

9. A BERT-like model has no self-attention sub-layers. (True/False)

   False. BERT has self-attention layers.

10. Data collators are helpful for...