Implementation of Megatron-LM

In this section, we will briefly discuss how to use Megatron-LM. For more details, you can read the official user manual of Megatron-LM here: https://github.com/NVIDIA/Megatron-LM. Here are the steps we need to follow:

1. To use Megatron-LM, we first need to install some pre-trained checkpoints, as follows:

   # Download checkpoints
   # Terminal
   wget --content-disposition \
       models/nvidia/megatron_lm_345m/ \
       versions/ \
       v0.1/zip \
       -0 \
       megatron_lm_345m_v0.1.zip

2. After that, you can use megatron-lm to preprocess the data, as follows:

   # pre-process data
   python3 preprocess_data.py \
           --input xxx.json \
           --output-prefix my-model \
           --vocab bert-vercab.txt \
          ...