RAG-Driven Generative AI: Build custom retrieval augmented generation pipelines with LlamaIndex, Deep Lake, and Pinecone

Denis Rothman

$43.99

4.5 (13 Ratings)

Paperback Sep 2024 334 pages 1st Edition

Denis Rothman

$43.99

4.5 (13 Ratings)

Paperback Sep 2024 334 pages 1st Edition

What do you get with Print?

Instant access to your digital eBook copy whilst your Print order is Shipped

Colour book shipped to your preferred address

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

View table of contents

Preview Book

RAG-Driven Generative AI

RAG Embedding Vector Stores with Deep Lake and OpenAI

There will come a point in the execution of your project where complexity is unavoidable when implementing RAG-driven generative AI. Embeddings transform bulky structured or unstructured texts into compact, high-dimensional vectors that capture their semantic essence, enabling faster and more efficient information retrieval. However, we will inevitably be faced with a storage issue as the creation and storage of document embeddings become necessary when managing increasingly large datasets. You could ask the question at this point, why not use keywords instead of embeddings? And the answer is simple: although embeddings require more storage space, they capture the deeper semantic meanings of texts, with more nuanced and context-aware retrieval compared to the rigid and often-matched keywords. This results in better, more pertinent retrievals. Hence, our option is to turn to vector stores in which embeddings are organized and rapidly...

Evaluating the output with cosine similarity

In this section, we will implement cosine similarity to measure the similarity between user input and the generative AI model’s output. We will also measure the augmented user input with the generative AI model’s output. Let’s first define a cosine similarity function:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
def calculate_cosine_similarity(text1, text2):
    vectorizer = TfidfVectorizer()
    tfidf = vectorizer.fit_transform([text1, text2])
    similarity = cosine_similarity(tfidf[0:1], tfidf[1:2])
    return similarity[0][0]

Then, let’s calculate a score that measures the similarity between the user prompt and GPT-4’s response:

similarity_score = calculate_cosine_similarity(user_prompt, gpt4_response)
print(f"Cosine Similarity Score: {similarity_score:.3f}")

The score is low, although the output seemed...

Download Code

Key benefits

Implement RAG’s traceable outputs, linking each response to its source document to build reliable multimodal conversational agents
Deliver accurate generative AI models in pipelines integrating RAG, real-time human feedback improvements, and knowledge graphs
Balance cost and performance between dynamic retrieval datasets and fine-tuning static data

Description

RAG-Driven Generative AI provides a roadmap for building effective LLM, computer vision, and generative AI systems that balance performance and costs. This book offers a detailed exploration of RAG and how to design, manage, and control multimodal AI pipelines. By connecting outputs to traceable source documents, RAG improves output accuracy and contextual relevance, offering a dynamic approach to managing large volumes of information. This AI book shows you how to build a RAG framework, providing practical knowledge on vector stores, chunking, indexing, and ranking. You’ll discover techniques to optimize your project’s performance and better understand your data, including using adaptive RAG and human feedback to refine retrieval accuracy, balancing RAG with fine-tuning, implementing dynamic RAG to enhance real-time decision-making, and visualizing complex data with knowledge graphs. You’ll be exposed to a hands-on blend of frameworks like LlamaIndex and Deep Lake, vector databases such as Pinecone and Chroma, and models from Hugging Face and OpenAI. By the end of this book, you will have acquired the skills to implement intelligent solutions, keeping you competitive in fields from production to customer service across any project.

Who is this book for?

This book is ideal for data scientists, AI engineers, machine learning engineers, and MLOps engineers. If you are a solutions architect, software developer, product manager, or project manager looking to enhance the decision-making process of building RAG applications, then you’ll find this book useful.

What you will learn

Scale RAG pipelines to handle large datasets efficiently
Employ techniques that minimize hallucinations and ensure accurate responses
Implement indexing techniques to improve AI accuracy with traceable and transparent outputs
Customize and scale RAG-driven generative AI systems across domains
Find out how to use Deep Lake and Pinecone for efficient and fast data retrieval
Control and build robust generative AI systems grounded in real-world data
Combine text and image data for richer, more informative AI responses

Estimated delivery fee Deliver to Egypt

Standard delivery 10 - 13 business days

$12.95

Premium delivery 3 - 6 business days

$34.95

(Includes tracking information)

Product Details

Publication date, Length, Edition, Language, ISBN-13

Publication date : Sep 30, 2024

Length: 334 pages

Edition : 1st

Language : English

ISBN-13 : 9781836200918

Vendor :

Facebook , Docker , OpenAI

Category :

Data

Languages :

Python

Concepts :

GPT/LLMs

Tools :

Docker

What do you get with Print?

Instant access to your digital eBook copy whilst your Print order is Shipped

Colour book shipped to your preferred address

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

Frequently bought together

Unlocking Data with Generative AI and RAG

$39.99

$43.99

$49.99

Total $ 133.97

FAQs

What is the delivery time and cost of print book?

Shipping Details

USA:

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.

Unfortunately, due to several restrictions, we are unable to ship to the following countries:

Afghanistan
American Samoa
Belarus
Brunei Darussalam
Central African Republic
The Democratic Republic of Congo
Eritrea
Guinea-bissau
Iran
Lebanon
Libiya Arab Jamahriya
Somalia
Sudan
Russian Federation
Syrian Arab Republic
Ukraine
Venezuela

What is custom duty/charge?

Customs duty are charges levied on goods when they cross international borders. It is a tax that is imposed on imported goods. These duties are charged by special authorities and bodies created by local governments and are meant to protect local industries, economies, and businesses.

Do I have to pay customs charges for the print book order?

The orders shipped to the countries that are listed under EU27 will not bear custom charges. They are paid by Packt as part of the order.

List of EU27 countries: www.gov.uk/eu-eea:

A custom duty or localized taxes may be applicable on the shipment and would be charged by the recipient country outside of the EU27 which should be paid by the customer and these duties are not included in the shipping charges been charged on the order.

How do I know my custom duty charges?

The amount of duty payable varies greatly depending on the imported goods, the country of origin and several other factors like the total invoice amount or dimensions like weight, and other such criteria applicable in your country.

For example:

If you live in Mexico, and the declared value of your ordered items is over $ 50, for you to receive a package, you will have to pay additional import tax of 19% which will be $ 9.50 to the courier service.
Whereas if you live in Turkey, and the declared value of your ordered items is over € 22, for you to receive a package, you will have to pay additional import tax of 18% which will be € 3.96 to the courier service.

How can I cancel my order?

Cancellation Policy for Published Printed Books:

You can cancel any order within 1 hour of placing the order. Simply contact customercare@packt.com with your order details or payment transaction id. If your order has already started the shipment process, we will do our best to stop it. However, if it is already on the way to you then when you receive it, you can contact us at customercare@packt.com using the returns and refund process.

Please understand that Packt Publishing cannot provide refunds or cancel any order except for the cases described in our Return Policy (i.e. Packt Publishing agrees to replace your printed book because it arrives damaged or material defect in book), Packt Publishing will not accept returns.

What is your returns and refunds policy?

Return Policy:

We want you to be happy with your purchase from Packtpub.com. We will not hassle you with returning print books to us. If the print book you receive from us is incorrect, damaged, doesn't work or is unacceptably late, please contact Customer Relations Team on customercare@packt.com with the order number and issue details as explained below:

If you ordered (eBook, Video or Print Book) incorrectly or accidentally, please contact Customer Relations Team on customercare@packt.com within one hour of placing the order and we will replace/refund you the item cost.
Sadly, if your eBook or Video file is faulty or a fault occurs during the eBook or Video being made available to you, i.e. during download then you should contact Customer Relations Team within 14 days of purchase on customercare@packt.com who will be able to resolve this issue for you.
You will have a choice of replacement or refund of the problem items.(damaged, defective or incorrect)
Once Customer Care Team confirms that you will be refunded, you should receive the refund within 10 to 12 working days.
If you are only requesting a refund of one book from a multiple order, then we will refund you the appropriate single item.
Where the items were shipped under a free shipping offer, there will be no shipping costs to refund.

On the off chance your printed book arrives damaged, with book material defect, contact our Customer Relation Team on customercare@packt.com within 14 days of receipt of the book with appropriate evidence of damage and we will work with you to secure a replacement copy, if necessary. Please note that each printed book you order from us is individually made by Packt's professional book-printing partner which is on a print-on-demand basis.

What tax is charged?

Currently, no tax is charged on the purchase of any print book (subject to change based on the laws and regulations). A localized VAT fee is charged only to our European and UK customers on eBooks, Video and subscriptions that they buy. GST is charged to Indian customers for eBooks and video purchases.

What payment methods can I use?

You can pay with the following card types:

Visa Debit
Visa Credit
MasterCard
PayPal

What is the delivery time and cost of print books?

Shipping Details

USA:

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Unfortunately, due to several restrictions, we are unable to ship to the following countries:

Afghanistan
American Samoa
Belarus
Brunei Darussalam
Central African Republic
The Democratic Republic of Congo
Eritrea
Guinea-bissau
Iran
Lebanon
Libiya Arab Jamahriya
Somalia
Sudan
Russian Federation
Syrian Arab Republic
Ukraine
Venezuela

Filter reviews by

All

Feefo verified reviews

Amazon verified reviews

Melvin Ng Nov 21, 2024

Nice book

Feefo Verified review

dr t Oct 04, 2024

Rothman has once again delivered something exceptional with RAG-Driven Generative AI. As expected from Rothman, this book shines in its ability to make complex topics accessible and practical, making it a standout in the growing literature on RAG systems. If you're looking for one of the best resources on RAG, packed with Python code and real-world applications, this book will not let you down.For readers keen to get hands-on, the book does not disappoint. Rothman provides a wealth of Python code throughout, with step-by-step examples that make it easy to follow along and implement RAG-driven solutions. Each chapter concludes with questions to test your understanding, reinforcing key concepts and ensuring that you grasp the material before moving on. For beginners and experienced practitioners alike, this interactive approach adds immense value to the learning experience.Chapter 4, Building a RAG Pipeline, is particularly valuable, offering clear instructions on how to build an end-to-end RAG system. The chapter walks readers through the process of designing a robust RAG pipeline. In addition, Rothman explores cutting-edge tools such as LlamaIndex, Deep Lake, and OpenAI to illustrate how to leverage them effectively for RAG-based projects. The comprehensive nature of this chapter makes it an essential guide for anyone looking to develop RAG systems from scratch or optimise existing ones.However, the most enlightening part of the book for this reader was Chapter 5: Boosting RAG Performance with Expert Human Feedback. This chapter delves into the creation of an adaptive RAG system that can evolve based on user feedback. Rothman guides readers through building a hybrid adaptive RAG program in Python on Google Colab. This hands-on project not only gives readers a solid grasp of adaptive RAG processes but also demonstrates how to adjust a system when predefined models fail to meet user expectations. Rothman goes further to show how human feedback, gathered through user rankings, can be integrated to fine-tune RAG systems, ensuring that the AI continues to meet users' needs. The chapter concludes with the implementation of an automated ranking system to enhance the generative model's performance, making it highly applicable to real-world business settings.In conclusion, RAG-Driven Generative AI is a must-read for anyone involved with LLMs. Rothman has delivered an insightful, practical, and highly recommended resource for anyone looking to explore RAG systems. Highly recommended.

Amazon Verified review

Jorge Deflon Oct 10, 2024

I have been reading this new book on generative artificial intelligence complemented with RAG (Retrieval-Augmented Generation) and I find it quite useful and interesting.LLM models are advanced artificial intelligence systems designed to process and generate human language.They are trained with enormous amounts of text from several sources, to understand and respond coherently to a wide variety of questions and requests, but this also carries the disadvantage that they may not have the most relevant information for an organization, since it was not available when the model was trained, either due to time or confidentiality issues.Retrieval enhanced generation (RAG) is the process of optimizing the output so that it references an personalized knowledge base before generating a response.This allows the GAI to produce more useful and reliable responses to the organization's users.This book is one of the most complete and up-to-date references on how to use RAG techniques to improve the responses that GAI tools provide to organizational users.The book contains many examples on how use the different types of RAG, including the necessary code to incorporate it into your projects quickly and efficiently.Highly recommended for all practitioners, developers, and students of the topic of generative artificial intelligence.

Subhayan Roy Oct 11, 2024

RAG being in the forefront of Gen AI LLM models is a highly sought after skill or knowledge to have.This book covers the theory part of RAG, vectorization, Vector databases.Yet what I found most fascinating was the code snippets, applications that you can directly use in your GenAI application with a bit of modification.Just one advice be clear on Transformer and language models before learning RAG.For this I would recommend Denis's other book Transformers for NLP.

Siddhartha Vemuganti Oct 15, 2024

Denis Rothman's "RAG-Driven Gen AI" offers a comprehensive exploration of Retrieval-Augmented Generation systems, addressing a critical need in the rapidly evolving field of artificial intelligence. This book stands out for its practical approach, bridging the gap between theoretical concepts and real-world applications.Rothman's writing style is accessible yet thorough, guiding readers from foundational principles to advanced implementations of RAG systems. The book's structure feels well-considered, allowing readers to build their understanding progressively. While it assumes some prior knowledge of machine learning and Python, making it less suitable for complete beginners, it offers valuable insights for software engineers, developers, and data scientists looking to expand their AI toolkit.One of the book's strengths lies in its diverse range of practical examples. By covering applications from drone technology to customer retention, Rothman effectively demonstrates the versatility of RAG systems. The chapter on multimodal RAG for drone technology is particularly intriguing, opening up new possibilities that many readers might not have previously considered.A standout feature is the book's attention to often-overlooked aspects of AI development, such as software versioning and package management. Rothman's detailed guidance on version control and dependency management addresses real challenges faced by practitioners, potentially saving readers significant time and frustration.The hands-on approach, complete with projects and source code, transforms the book from a mere reference into a practical learning tool. Rothman doesn't shy away from discussing performance optimization and cost management – crucial considerations for implementing AI solutions in production environments.However, readers should be aware that the rapid pace of AI advancement may necessitate supplementing this book with current research and developments. Some cutting-edge concepts discussed may evolve quickly."RAG-Driven Gen AI" serves as a valuable resource for those looking to understand and implement RAG systems. While it may not be the only book you'll need on the subject, it provides a solid foundation and practical insights that many readers will find useful. Rothman's work effectively captures the current state of RAG technology while offering guidance that should remain relevant as the field continues to evolve.For professionals aiming to leverage the power of RAG systems or enhance their AI capabilities, this book is a worthwhile addition to their technical library. It offers a balanced mix of theoretical understanding and practical application, making it a useful companion for those navigating the complex landscape of modern AI development.

RAG-Driven Generative AI: Build custom retrieval augmented generation pipelines with LlamaIndex, Deep Lake, and Pinecone

What do you get with Print?

RAG-Driven Generative AI

RAG Embedding Vector Stores with Deep Lake and OpenAI

From raw data to embeddings in vector stores

Organizing RAG in a pipeline

A RAG-driven generative AI pipeline

Building a RAG pipeline

Setting up the environment

Evaluating the output with cosine similarity

Summary

Questions

References

Further reading

Join our community on Discord

Page 1 of 10

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

About the author

FAQs

RAG-Driven Generative AI: Build custom retrieval augmented generation pipelines with LlamaIndex, Deep Lake, and Pinecone

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

About the author

FAQs