cURL will be our tool of choice that we will use to communicate with Elasticsearch. Elasticsearch follows a REST-like protocol for its exposed web API. Some of its features are as follows:

To specify the HTTP method in cURL, you can use the -X option, for example, CURL -X POST http://localhost/. JSON is the data format used to communicate with Elasticsearch. To specify the data in cURL, we can specify it in the following forms:

curl -X PUT "localhost:9200/news" -d '{
"settings": {
"index": {
"number_of_shards": 2,
"number_of_replicas": 1
}
}
}'

curl -X PUT "localhost:9200/news/public/_mapping" -d '{
"public" :{
"properties" :{
"Title" : {"type" : "string" },
"Content": {"type" : "string" },
"DOP": {"type" : "date" }
}
}
}'

Analyzers constitute an important part of indexing. To understand what analyzers do, let's consider three documents:

Here, terms such as This, is, as well as and are not relevant keywords. The chances of someone wanting to search for such words are very less, as these words don't contribute to the facts or context of the document. Hence, it's safe to avoid these words while indexing or rather you should avoid making these words searchable.

So, the tokenization would be as follows:

Words such as the, or, as well as and are referred to as stop words. In most cases, these are for grammatical support and the chances that someone will search based on these words are slim. Also, the analysis and removal of stop words is very much language dependent. The process of selecting/transforming the searchable tokens from a document while indexing is called analyzing. The module that facilitates this is called an analyzer. The analyzer we just discussed is a stop word analyzer. By applying the right analyzer, you can minimize the number of searchable tokens and hence get better performance results.

There are three stages through which you can perform an analysis:

Here is a flowchart that depicts this process:

It should be noted that any number of such components can be incorporated in each stage. A combination of these components is called an analyzer. To create an analyzer out of the existing components, all we need to do is add the configuration to our Elasticsearch configuration file.

index :
analysis :
analyzer :
myCustomAnalyzer :
tokenizer : smallLetter
filter : [lowercase, stopWord]
char_filter : [html_strip]
tokenizer :
smallLetter:
type : standard
max_token_length : 900
filter :
stopWord:
type : stop
stopwords : ["are" , "the" , "is"]

curl -X PUT "http://localhost:9200/wiki" -d '{   
  "index" : { 
    "number_of_shards" : 4, 
    "number_of_replicas" : 1 ,
    "analysis":{      
      "analyzer":{         
        "content" : {
          "type" : "custom",
          "tokenizer" : "standard", 
          "filter" : ["lowercase" , "stop" , "kstem"],
          "char_filter" : ["html_strip"]
        }
      }
    }
  }
  
}'

import urllib2
import json
import sys

link = sys.argv[2]
htmlObj = { "link" : link , 
    "Author" : "anonymous" ,
    "timestamp" : "09-02-2014 14:16:00",
    "Title" : sys.argv[1]
     }
response = urllib2.urlopen(link)
htmlObj['html'] = response.read()
print json.dumps(htmlObj ,  indent=4)

Python json_generator.py https://en.wikipedia.org/wiki/France > France.json'.

curl -X PUT "http://localhost:9200/wiki" -d '{   
      "index" : { 
    "number_of_shards" : 4, 
    "number_of_replicas" : 1 ,
        "analysis":{      
          "analyzer":{         
        "content" : {
          "type" : "custom",
          "tokenizer" : "standard", 
          "filter" : ["lowercase" , "stop" , "kstem"],
          "char_filter" : ["html_strip"]
        }
          }
        }
      }
  
}'

curl -X PUT "http://localhost:9200/wiki/articles/_mapping" -d '{
  "articles" :{
    "_all" : {"enabled" : true },
    "properties" :{
    "Title" : { "type" : "string" , "Analyzer":"content" ,  "include_in_all" : true},
    "link" : { "type" : "string" ,  "include_in_all" : false , "index" : "no" },
    "Author" : { "type" : "string" , "include_in_all" : false   },
    "timestamp" : { "type" : "date", "format" : "dd-MM-yyyy HH:mm:ss" , "include_in_all" : false },
    "html" : { "type" : "string" ,"Analyzer":"content" ,  "include_in_all" : true }
    }
  }
}'

curl -XPOST 'http://localhost:9200/wiki/articles/' -d @India.json

We added some documents to the index that we created. Now, let's examine some ways to query our data. Elasticsearch provides many types of queries to query our indexed documents. Of all the ones available, the simple query string query is a great place to start. The main advantage of this query is that it will never throw an exception. Also, a simple query string query discards the invalid parts of the query.

It mostly covers what is expected from most of the search engines. It takes OR of all the terms present in the query text, though we can change this behavior to AND. Also, it recognizes all Boolean keywords in the query text and performs the search accordingly. For details, you can look through http://lucene.apache.org/core/2_9_4/queryparsersyntax.html.

To query an Elasticsearch index, we must create a JSON query. A simple JSON query is shown here:

{
"query": {
    "simple_query_string": {
      "query": "sms",
      "fields": [
        "_all"
      ]
    }
  }

The screenshot of how a query is passed and the response is received in the head UI is shown as follows:

The explanation of the field's result is as follows:

When we searched for a record, what we got was its actual data or _source. However, this information is not what we actually need in search results. Instead, we want to extract the text out of the content, which helps the users to better understand the context where the text was matched in the document. For example, say the user searched for the word cochin, he would like to check whether the document speaks about the city Cochin or the cochin bank in Japan. Seeing other words around the word cochin will further help the user to judge whether that is the document he/she is searching for. Elasticsearch provides you with fragments of text on request for the highlighted text. Each fragment has the matched text and some words around it. As there can be any number of matched queries in the same document, you would be provided an array of fragments per document, where each fragment would contain the context of the matched query.

Here is how we ask Elasticsearch to provide the highlighted text:

{
"query" : {...},
"highlight" : {
"fields" : {
"Content" : {}
}
}
}

Under fields, you need to specify which all fields' highlighted text is required by you. In this example, we require the Content field.

Now, let's see another awesome feature that Elasticsearch offers. You would have noticed in Google search that the matched text in the highlighted fragments is shown in bold. Elasticsearch provides support for this as follows:

{
"query" : {...},
"highlight" : {
"pre_tags" : ["<b>"],
"post_tags" : ["</b>"],
"fields" : {
"Content" : {}
}
}
}

Here, you can mention the pre tag and post tag. To get the matched text in bold, simply input pre tag as <b> and post tag as </b>. By default, the <em> </em> tags are provided. The maximum number of fragments and maximum number of words per fragment are also configurable.

While searching, users can't view all the results at once. They like to see one batch at a time. Usually, a single batch contains 10 matched documents, as in Google search results, where each page contains 10 search results. This also gives us an advantage over the search engine as it need not send all the results back at once. The following is how we use pagination in Elasticsearch. Let's say that we are interested in seeing only five results at a time, then to get the first page, we have to use the following parameters:

Also, it should be noted that the total number of pages can be calculated from count/_size. Sample query for the page 5 of the search result where we show 5 results at a time:

{
"from" : 4 ,
"size" : 5,
"query": {… }  }

This is how the complete query looks, which enables pagination and highlighting:

{
  "from": 0,
  "size": 10,
  "query": {
    "simple_query_string": {
      "query": "china",
      "fields": [
        "_all"
      ]
    }
  },
  "highlight": {
    "fields": {
      "html": {
        "pre_tags": [
          "<p>"
        ],
        "post_tags": [
          "</p>"
        ],
        "fragment_size": 10,
        "number_of_fragments": 3
      }
    }
  }
}

The head UI explained

When you open the head page, you see a UI that lists all the indexes and all the information related to it. Also, by looking at the tabs to the left, you know how well your cluster is doing, as shown in the following figure:

Now, take the Browser tab in the head UI. You will see all the feeds you index here. Note that it shows only the first 10 indexed feeds.

Now, on selecting one of your feeds, a nice model window appear, showing you the following view:

In this chapter, we looked at how we can deploy Elasticsearch. We had a quick look at of how to set an analyzer and index some documents. Then, we attempted to search for a document we indexed. We will look at how pagination and highlighting work in later sections of this book.

Kick starting Elasticsearch is much easier than any other open source projects. It ships with the best possible configurations, which make the process of starting this easy, and it ships with the most optimistic settings for performance. Hence, the initial learning curve on the user side is reduced. We went through a getting started that was easy; and discussed some of the architectural choices, which make this application truly distributed.

Though Elasticsearch head is a good tool to interact with Elasticsearch. There are other choices, such as Sense (packed with Elasticsearch Marvel), KOPF, and so on, which can also be used for the same purpose. There is a wide variety of ways in which we can use analyzers to improve a user's search experience. A separate chapter is dedicated to this in this book.

In the next chapter, you will learn how you can effectively use Elasticsearch to build an e-commerce application. Elasticsearch is a natural fit to build an e-commerce application. Search over structured and unstructured data, pagination, scoring, aggregation, filtering, and highlighting makes Elasticsearch an ideal backend for e-commerce-related applications.