This chapter has the following technical requirements:

• Hardware: A desktop or laptop computer with a minimum of 4 GB of RAM; 8 GB is suggested
• Operating system: A Unix-like operating system such as macOS, or any Linux-based distribution, such as Ubuntu
• Programming Language: Python 3.7 or higher, and Python Package Installer, or pip

This section will guide you through the fundamentals of symbolic search systems, the different types of search applications, and their importance. This is followed by a brief description of how the symbolic search system works, with some code written in Python. Last but not least, we’ll summarize the pros and cons of the traditional symbolic search versus neural search. This will help us to understand how a neural search can better bridge the gap between a user’s intent and the retrieved documents.

Exploring various data types and search scenarios

In today’s society, governments, enterprises, and individuals create a huge amount of data by using various platforms every day. We live in the era of big data, where things such as texts, images, videos, and audio files play a significant role in society and the fulfillment of daily tasks.

Generally speaking, there are three types of data:

• Structured data: This includes data that is logically expressed and realized by a two-dimensional table structure. Structured data strictly follows a specific data format and length specifications and is mainly stored and managed using relational databases.
• Unstructured data: This has neither a regular or complete structure nor a predefined data model. This type of data is not appropriately managed by representing the data using a two-dimensional logical table used in databases. This includes office documents, text, pictures, hypertext markup language (HTML), various reports, images, and audio and video information in all formats.
• Semi-structured data: This falls somewhere between structured and unstructured data. It includes log files, Extensible Markup Language (XML), and Javascript Object Notation (JSON). Semi-structured data does not conform to the data model structure associated with relational databases or other data tables, but it contains relevant tags that can be used to separate semantic elements that are used to stratify records and fields.

Search indices are widely used to hunt for unstructured and semi-structured data within a massive data collection to meet the information needs of users. Based on the levels and applications of the document collection, searches can be further divided into three types: web search, enterprise search, and personal search.

In a web search, the search engine first needs to index hundreds of millions of documents. The search results are then returned to users in an efficient manner while the system is continuously optimized. Typical examples of web search applications are Google, Bing, and Baidu.

In addition to web search, as a software development engineer, you are likely to encounter enterprise and personal search operations. In enterprise search scenarios, the search engine indexes internal documents of an enterprise to serve the employees and customers of the business, such as an internal patent search index of a company, or the search index of a music platform, such as SoundCloud.

If you are developing an email application and intend to allow users to search for historical emails, this constitutes a typical example of a personal search. This book focuses on enterprise and personal types of search operations.

Important Note

Make sure you understand the difference between search and match. Search, in most cases, is done in documents organized in an unstructured or semi-structured format, while match (such as an SQL-like query) takes place on structured data, such as tabular data.

As for different data types, the concept of modality plays an important role in a search system. Modality refers to the form of information such as text, images, video, and audio files. Cross-modality search (also known as cross-media search) refers to retrieving samples from different modes with similar semantics by exploring the relationship between different modalities and employing a certain modal sample.

For example, when we enter a keyword in an email inbox application, we can find the appropriate email returns as a result of a unimodal search – searching text by text. When you enter a keyword on a page for image retrieval, the search engine will return appropriate images as a result of a cross-modal search, searching images by text.

Of course, a unimodal search is not limited to searching text by text. The app known as Shazam, which is popular in the App Store, helps users to identify music and returns a track’s title to users in a short time. This can be seen as an application of unimodal search. Here, the concept of modality no longer refers to text, but to audio. On Pinterest, users can locate similar images through an image search, where the modality refers to an image. Likewise, the scope of a cross-modal search covers far more than searching for images by text.

Let’s consider this from another perspective. Is it possible for us to search across multiple modalities? Of course, the answer is “Yes!” Imagine a search scenario where a user uploads a photo of clothes and wants to look for similar types of clothing (we usually call this type of application “shop the look”), and at the same time enters a paragraph that describes the clothes in the search box to improve the accuracy of the search. In this way, our search keywords span two modalities (text and images). We refer to this search scenario as a multi-modal search.

Now that we have a grasp of the concept of modality, we will elaborate on the working principles, advantages, and disadvantages of symbolic search systems. By the end of this section, you will understand, that symbolic search systems cannot deal with different modalities.

How does the traditional search system work?

As a developer, you may have used Elasticsearch or Apache Solr to build a search system in web applications. These two widely used search frameworks were developed based on Apache Lucene. We’ll take Lucene as a case in point to introduce the components of a search system. Imagine you intend to search for a keyword in thousands of text documents (txt). How will you complete this task?

The easiest solution is to traverse all text documents from a path and read through the contents of these documents. If the keyword is in the file, the name of the document will be returned:

# src/chapter-1/sequential_match.py
import os
import glob
dir_path = os.path.dirname(os.path.realpath(__file__))
def match_sequentially():
    matches = []
    query = 'hello jina'
    txt_files = glob.glob(f'{dir_path}/resources/*.txt')
    for txt_file in txt_files: 
        with open(txt_file, 'r') as f:
            if query in f.read(): 
                matches.append(txt_file)
    return matches
if __name__ == '__main__':
    matches = match_sequentially()
    print(matches)

The code fulfills the simplest search function by traversing all files with the extension .txt in the current directory and then opening those files in turn. If the keyword hello jina used for the query is available, the filename will be printed with all the matching files. Although these lines of code allow you to conduct a basic search, the process has many flaws:

• Poor scalability: In a production environment, there may be millions of files to be retrieved. Meanwhile, users of the retrieval system expect to obtain retrieval results in the shortest possible time, posing stringent requirements for the performance of the search system.
• Lack of a relevance measurement: The code helps you achieve the most basic Boolean retrieval, which is to return the result of a match or mismatch. In a real-world scenario, users need a score to measure the relevance degree from a search system that is sorted in descending order, with more relevant files being returned to users first. Obviously, the aforementioned code snippets are unable to fulfill this function.

To address these issues, we need to index the files to be retrieved. Indexing refers to a process of converting a file type that allows a rapid search and skipping the continuous scanning of all files.

As an important part of our daily lives, indexing is comparable to consulting a dictionary and visiting a library. We’ll use the most widely used search library, Lucene, to illustrate the idea.

Lucene Core (https://lucene.apache.org/) is a Java library providing powerful indexing and search features, as well as spellchecking, hit highlighting, and advanced analysis/tokenization capabilities. Apache Lucene sets the standard for search and indexing performance. It is the search core of both Apache Solr and Elasticsearch.

In Lucene, after all collections of files to be retrieved are loaded, you may extract texts from such files and convert them to Lucene Documents, which generally contain the title, body, abstract, author, and URL of a file.

Next, your file will be analyzed by Lucene’s text analyzer, which generally includes the following processes:

Tokenizer: This splits the raw input paragraphs into tokens that cannot be further decomposed.

Decomposing compound words: In languages such as German, words composed of two or more tokens are called compound words.

Spell correction: Lucene allows users to conduct spellchecking to enhance the accuracy of retrieval.

Synonym analysis: This enables users to manually add synonyms in Lucene to improve the recall rate of the search system (note: the accuracy rate and recall rate will be elaborated upon shortly).

Stemming and lemmatization: The former enables users to derive the root by removing the suffix of a word (for example, play, the root form, is derived from the words plays, playing, and played), while the latter helps users convert words into basic forms, such as is, are, and been, which are converted to be.

Let’s attempt to preprocess some texts using NLTK.

Important Note

NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources.

First, install a Python package called nltk with this command:

pip install nltk
python -m nltk.downloader 'punkt'

We preprocess the text Jina is a neural search framework built with cutting-edge technology called deep learning:

import nltk
sentence = 'Jina is a neural search framework built with cutting-edge technology called deep learning'
def tokenize_and_stem():
    tokens = nltk.word_tokenize(sentence)
    stemmer = nltk.stem.porter.PorterStemmer()
    stemmed_tokens = [stemmer.stem(token) for token in 
                     tokens]
    return stemmed_tokens
if __name__ == '__main__':
    tokens = tokenize_and_stem()
    print(tokens)

This code enables us to carry out two operations on a sentence: tokenizing and stemming. The results of each are printed respectively. The raw input strings are parsed into a list of strings in Python, and finally each parsed token is lemmatized to its basic form. For instance, cutting and called are respectively converted to cut and call. For more operations, please refer to the official documentation of NLTK (https://www.nltk.org/).

After files are processed with the Lucene Document, the clean files will be indexed. Generally, in a traditional search system, all files are indexed using an inverted index. An inverted index (also referred to as a postings file or inverted file) is an index data structure storing a map of content, such as words or numbers, to its locations in a database file, or in a document or a set of documents.

Simply put, an inverted index consists of two parts: a term dictionary, and postings.

Tokens, their IDs, and the document frequency (the frequency of such tokens appearing in the entire collection of documents to be retrieved) are stored in the term dictionary. A collection of all tokens is called a vocabulary. All tokens are sorted in alphabetical order in the dictionary.

In the postings, we save the token ID and the document IDs where the token occurred. Assuming that in the aforesaid example, the token jina from our query keyword hello jina appears three times in the entire collection of documents (in 1.txt, 3.txt, and 11.txt), then the token is “jina” and the document frequency is 3. Meanwhile, the names of the three text documents, 1.txt, 3.txt, and 11.txt, are saved in the posting. Then, the indexing of the text file is completed as shown in the following figure:

Figure 1.1 – Data structure of inverted index

When a user makes a query, keywords used for the query are generally shorter than the collection of documents to be retrieved. Lucene can perform the same preprocessing for such keywords (such as tokenization, decomposition, and spelling correction).

The processed tokens are mapped to the postings through the term dictionary in the inverted index so that matched files can be quickly found. Finally, Lucene’s scoring starts to work and scores each related file discovered according to a vector space model. Our index file is stored in an inverted index, which may be represented as a vector.

Assuming that our query keyword is jina, we map it to the vector of the inverted index and have it represented by - when it does not appear in the file; then the query vector [-,'jina',-,-, ...] can be obtained. This is how we represent a query, as a vector space model, in a traditional search engine.

Figure 1.2 – Term occurrence in the vector space model

Next, in order to derive the ranking, we need to numerically represent the token of the space vector model. Generally, tf-idf is regarded as a simple approach.

With this algorithm, we grant a higher weight to any token that appears relatively frequently. If such a token appears multiple times in many documents, we believe that the token is weakly representative, and the weight of the token will be reduced again. If the token does not appear in the documents, its weight is 0.

In Lucene, an algorithm called bm-25 is employed more frequently, which further optimizes tf-idf. After numerical calculation, the vector is expressed as follows:

Figure 1.3 – Vector space representation

As shown in the preceding figure, because the word a appears too frequently, it appears in document 1 and document 2 and has a low weight score. The token jina, a relatively uncommon word (appearing in document 2), has been granted a higher weight.

In the query vector, because the query keyword only has one word, jina, its weight is set as 1 and the weights of other tokens that do not appear are set as 0. Afterward, we multiply the query vector and the document vector element by element and add up the results to obtain the score of each document corresponding to the query keyword. Then, reverse sorting is performed so that the sorted documents can be returned to the user according to the score sorted in an inverted order (from high to low scores).

In short, if the keyword used for a query appears more frequently in a particular file and less frequently in the vocabulary file, its relative score will be higher and returned to the user with a higher priority. Of course, Lucene also grants different weights to various parts of a file. For example, the title and keywords of the file will have a higher scoring weight than the body would. Given the fact this book is about neural search, this aspect will not be elaborated upon further here.

Pros and cons of the traditional search system

In the previous section, we briefly revisited traditional symbolic search. Perhaps you have noticed that both the Lucene we introduced previously and the Lucene-based search frameworks, such as Elasticsearch and Solr, are based on text retrieval. This has quite a few advantages in the application scenarios of searching text by text:

Mature technology: Since research and development were done in 1999, the Lucene and Lucene-based search systems have existed for over 20 years and have been widely used in various web applications.

Easy integration: As users, developers of a web application do not need to have a deep understanding of Elasticsearch, Solr, or the operating logic of Lucene; only a small amount of code is required to integrate a high-performing, extensible search system into web applications.

Well-developed ecosystem: Thanks to the operation of Elastic Company, Elasticsearch has extended its search system functionality significantly. Currently, it is not only a search framework, but also a platform equipped with user management, a restful interface, data backup and restoration, and security management such as single sign-in, log audit, and other functions. Meanwhile, the Elasticsearch community has contributed a variety of plugins and integrations.

At the same time, you have probably realized that both Elasticsearch and Solr with Lucene at the core have unavoidable flaws.

In the previous section, we introduced the concept of modality. Lucene and Elasticsearch, which is built on top of it, are inherently unable to support cross-modal and multi-modal search options. Let’s take a moment to review the operating principle of Lucene, as Lucene has powered most of the search systems users are using on a daily basis. When texts are preprocessed in the first place, the search keyword must be text. When a data collection to be retrieved is preprocessed and indexed, likewise the index result is also the text stored in the inverted index.

In this way, the Lucene-based search platform can only rely on the text modality and retrieve data in the text modality. If objects to be retrieved are images, audio, or video files, how can they be found using a traditional search system? It is quite simple; two main methods are employed:

Manual tagging and adding metadata: For example, when a user uploads a song to a music platform, they may manually tag the author, album, music type, release time, and other data. Doing so ensures that users are able to retrieve music using text.

Hypothesis of the surrounding text: If an image, in the absence of user tagging, appears in an article, it will be assumed by the traditional search system to be more closely associated with its surrounding text. Accordingly, when a user’s query keyword matches the surrounding text of the image, the latter will be matched.

The essence of the two methods is to convert the document of non-text modality into a text modality so as to effectively use the current retrieval technology. However, this modal conversion process either relies on a large amount of manual tagging, or is done at the cost of query accuracy, which greatly undermines the user’s search experience.

Likewise, this type of search mode limits the user’s search habits to a keyword search and cannot be extended to a real cross-modal or even multi-modal search. For deeper insight into this issue, we may use a vector space to represent keywords of a paragraph and use another vector space to denote a text document to be retrieved. However, due to the restrictions of the technology back in the days when we had to rely on traditional search systems, we were unable to use the space vector to represent a piece of music, image, or video. It is also impossible to map two documents of different modalities to the same space vector to compare their similarities.

With the research and development on (statistical) machine learning techniques, more and more researchers and engineers have started to empower their search system using machine learning algorithms.