Chapter 4, Pretraining a RoBERTa Model from Scratch

1. RoBERTa uses a byte-level byte-pair encoding tokenizer. (True/False)

   True.

2. A trained Hugging Face tokenizer produces merges.txt and vocab.json. (True/False)

   True.

3. RoBERTa does not use token-type IDs. (True/False)

   True.

4. DistilBERT has 6 layers and 12 heads. (True/False)

   True.

5. A transformer model with 80 million parameters is enormous. (True/False)

   False. 80 million parameters is a small model.

6. We cannot train a tokenizer. (True/False)

   False. A tokenizer can be trained.

7. A BERT-like model has six decoder layers. (True/False)

   False. BERT contains six encoder layers, not decoder layers.

8. MLM predicts a word contained in a mask token in a sentence. (True/False)

   True.

9. A BERT-like model has no self-attention sublayers. (True/False)

   False. BERT has self...