Retrieval augmented generation (RAG)

In this recipe, we will see vector embeddings in action. RAG is a popular method of working with LLMs. Since these models are pretrained on widely available internet data, they do not have access to our personal data, and we cannot use the model as it is to ask questions about it. A way to overcome this is to use vector embeddings to represent our data. Then, we can compute cosine similarity between our data and the question and include the most similar piece of our data, together with the question – hence the name “retrieval augmented generation,” since we first retrieve relevant data by using cosine similarity and then generate text using the LLM.

Getting ready

We will use an IMDB dataset from Kaggle, which can be downloaded from https://www.kaggle.com/PromptCloudHQ/imdb-data and is also included in the book GitHub repo at https://github.com/PacktPublishing/Python-Natural-Language-Processing-Cookbook-Second-Edition/blob...