Implementing the original Transformer

The following code demonstrates how to implement a minimal transformer model for a Seq2Seq translation task, mainly translating English text to French. The code is structured into multiple sections, handling various aspects from data loading to model training and translation.

Data loading and preparation

Initially, the code loads a dataset and prepares it for training. The data is loaded from a CSV file, which is then split into English and French text. The text is limited to 100 characters for demonstration purposes to reduce training time. The CSV file includes a few thousand example data points and can be found in the book’s GitHub repository (https://github.com/PacktPublishing/Python-Generative-AI) along with the complete code:

import pandas as pd
import numpy as np
# Load demo data
data = pd.read_csv("./Chapter_3/data/en-fr_mini.csv")
# Separate English and French lexicons
EN_TEXT = data.en.to_numpy().tolist()
FR_TEXT...