Code lab 8.2 – Hybrid search with a custom function
The file you need to access from the GitHub repository is titled CHAPTER8-2_HYBRID_CUSTOM.ipynb
.
In this code lab, we are going to start with the notebook from Chapter 5: CHAPTER5-3_BLUE_TEAM_DEFENDS.ipynb
. Note that we are not using the Chapter 6 or 7 code, which has a lot of miscellaneous code we won’t use going forward. There is an added bonus in this code lab though; we are going to introduce some new elements that will carry us through the next couple of chapters, such as a new type of document loader for PDFs rather than web pages, a new larger document with more data to search, and a new text splitter. We will also clean out any code we no longer need as a result of these changes.
Once we have updated the code for these changes, we can focus on the task at hand, which is to use BM25 to generate our sparse vectors, combining those vectors with the dense vectors we have already used to form a hybrid search...