Vector Databases

Sometimes, data is rich with information and has a well-defined structure. If you know what you want, then this data is straightforward to work with in a modern database system. However, you often don’t know exactly what you need. Without specific search terms or phrases, you may not receive optimal search results. For example, you might not know the brand or name of your picky pet’s favorite food. In such complex cases, traditional information search and retrieval methods can fall short.

Modern AI research has given rise to a new class of methods that can encode the underlying semantic meaning of something instead of just its raw data. For example, AI models can understand that when you ask for the new action movie with that one actor who was also in the movie with green falling numbers, you’re asking for the latest John Wick film, which stars Keanu Reeves, who was also the star of The Matrix films.

To achieve this result, these methods convert their inputs into a numerical format called a vector embedding. Vector databases provide a means to efficiently store, organize, and search these vector representations. This makes vector databases valuable tools for retrieval tasks, which are common in AI applications. In this chapter, you will learn about vector search, the key concepts and algorithms associated with it, and the significance of vector databases. By the end of this chapter, you will understand the workings of graph connectivity and its application in architecture patterns such as RAG. You will also understand the best practices for building vector search systems.

This chapter will cover the following topics:

• Vector embeddings and similarity
• Nearest neighbor vector search
• The need for vector databases
• Case studies and real-world applications
• Vector search best practices