Understanding RAG – Basics and principles

Modern-day LLMs are impressive, but they have never seen your company’s private data (hopefully!). This means the ability of an LLM to help your company fully utilize its data is very limited. This very large barrier has given rise to the concept of RAG, where you are using the power and capabilities of the LLM but combining it with the knowledge and data contained within your company’s internal data repositories. This is the primary motivation for using RAG: to make new data available to the LLM and significantly increase the value you can extract from that data.

Beyond internal data, RAG is also useful in cases where the LLM has not been trained on the data, even if it is public, such as the most recent research papers or articles about a topic that is strategic to your company. In both cases, we are talking about data that was not present during the training of the LLM. You can have the latest LLM trained on the most tokens ever, but if that data was not present for training, then the LLM will be at a disadvantage in helping you reach your full productivity.

Ultimately, this highlights the fact that, for most organizations, it is a central need to connect new data to an LLM. RAG is the most popular paradigm for doing this. This book focuses on showing you how to set up a RAG application with your data, as well as how to get the most out of it in various situations. We intend to give you an in-depth understanding of RAG and its importance in leveraging an LLM within the context of a company’s private or specific data needs.

Now that you understand the basic motivations behind implementing RAG, let’s review some of the advantages of using it.

Advantages of RAG

Some of the potential advantages of using RAG include improved accuracy and relevance, customization, flexibility, and expanding the model’s knowledge beyond the training data. Let’s take a closer look:

• Improved accuracy and relevance: RAG can significantly enhance the accuracy and relevance of responses that are generated by LLMs. RAG fetches and incorporates specific information from a database or dataset, typically in real time, and ensures that the output is based on both the model’s pre-existing knowledge and the most current and relevant data that you are providing directly.
• Customization: RAG allows you to customize and adapt the model’s knowledge to your specific domain or use case. By pointing RAG to databases or datasets directly relevant to your application, you can tailor the model’s outputs so that they align closely with the information and style that matters most for your specific needs. This customization enables the model to provide more targeted and useful responses.
• Flexibility: RAG provides flexibility in terms of the data sources that the model can access. You can apply RAG to various structured and unstructured data, including databases, web pages, documents, and more. This flexibility allows you to leverage diverse information sources and combine them in novel ways to enhance the model’s capabilities. Additionally, you can update or swap out the data sources as needed, enabling the model to adapt to changing information landscapes.
• Expanding model knowledge beyond training data: LLMs are limited by the scope of their training data. RAG overcomes this limitation by enabling models to access and utilize information that was not included in their initial training sets. This effectively expands the knowledge base of the model without the need for retraining, making LLMs more versatile and adaptable to new domains or rapidly evolving topics.
• Removing hallucinations: The LLM is a key component within the RAG system. LLMs have the potential to provide wrong information, also known as hallucinations. These hallucinations can manifest in several ways, such as made-up facts, incorrect facts, or even nonsensical verbiage. Often, the hallucination is worded in a way that can be very convincing, causing it to be difficult to identify. A well-designed RAG application can remove hallucinations much more easily than when directly using an LLM.

With that, we’ve covered the key advantages of implementing RAG in your organization. Next, let’s discuss some of the challenges you might face.

Challenges of RAG

There are some challenges to using RAG as well, which include dependency on the quality of the internal data, the need for data manipulation and cleaning, computational overhead, more complex integrations, and the potential for information overload. Let’s review these challenges and gain a better understanding of how they impact RAG pipelines and what can be done about them:

• Dependency on data quality: When talking about how data can impact an AI model, the saying in data science circles is garbage in, garbage out. This means that if you give a model bad data, it will give you bad results. RAG is no different. The effectiveness of RAG is directly tied to the quality of the data it retrieves. If the underlying database or dataset contains outdated, biased, or inaccurate information, the outputs generated by RAG will likely suffer from the same issues.
• Need for data manipulation and cleaning: Data in the recesses of the company often has a lot of value to it, but it is not often in good, accessible shape. For example, data from PDF-based customer statements needs a lot of massaging so that it can be put into a format that can be useful to a RAG pipeline.
• Computational overhead: A RAG pipeline introduces a host of new computational steps into the response generation process, including data retrieval, processing, and integration. LLMs are getting faster every day, but even the fastest response can be more than a second, and some can take several seconds. If you combine that with other data processing steps, and possibly multiple LLM calls, the result can be a very significant increase in the time it takes to receive a response. This all leads to increased computational overhead, affecting the efficiency and scalability of the entire system. As with any other IT initiative, an organization must balance the benefits of enhanced accuracy and customization against the resource requirements and potential latency introduced by these additional processes.
• Data storage explosion; complexity in integration and maintenance: Traditionally, your data resides in a data source that’s queried in various ways to be made available to your internal and external systems. But with RAG, your data resides in multiple forms and locations, such as vectors in a vector database, that represent the same data, but in a different format. Add in the complexity of connecting these various data sources to LLMs and relevant technical mechanisms such as vector searches and you have a significant increase in complexity. This increased complexity can be resource-intensive. Maintaining this integration over time, especially as data sources evolve or expand, adds even more complexity and cost. Organizations need to invest in technical expertise and infrastructure to leverage RAG capabilities effectively while accounting for the rapid increase in complexities these systems bring with them.
• Potential for information overload: RAG-based systems can pull in too much information. It is just as important to implement mechanisms to address this issue as it is to handle times when not enough relevant information is found. Determining the relevance and importance of retrieved information to be included in the final output requires sophisticated filtering and ranking mechanisms. Without these, the quality of the generated content could be compromised by an excess of unnecessary or marginally relevant details.
• Hallucinations: While we listed removing hallucinations as an advantage of using RAG, hallucinations do pose one of the biggest challenges to RAG pipelines if they’re not dealt with properly. A well-designed RAG application must take measures to identify and remove hallucinations and undergo significant testing before the final output text is provided to the end user.
• High levels of complexity within RAG components: A typical RAG application tends to have a high level of complexity, with many components that need to be optimized for the overall application to function properly. The components can interact with each other in several ways, often with many more steps than the basic RAG pipeline you start with. Every component within the pipeline needs significant amounts of trials and testing, including your prompt design and engineering, the LLMs you use and how you use them, the various algorithms and their parameters for retrieval, the interface you use to access your RAG application, and numerous other aspects that you will need to add over the course of your development.

In this section, we explored the key advantages of implementing RAG in your organization, including improved accuracy and relevance, customization, flexibility, and the ability to expand the model’s knowledge beyond its initial training data. We also discussed some of the challenges you might face when deploying RAG, such as dependency on data quality, the need for data manipulation and cleaning, increased computational overhead, complexity in integration and maintenance, and the potential for information overload. Understanding these benefits and challenges provides a foundation for diving deeper into the core concepts and vocabulary used in RAG systems.

To understand the approaches we will introduce, you will need a good understanding of the vocabulary used to discuss these approaches. In the following section, we will familiarize ourselves with some of the foundational concepts so that you can better understand the various components and techniques involved in building effective RAG pipelines.