To follow and test the commands shown in this chapter, you must have a working Elasticsearch cluster installed on your system, as described in Chapter 1, Getting Started.

To simplify how you manage and execute these commands, I suggest that you install Kibana so that you have a more advanced environment to execute Elasticsearch queries.

If we consider the index as a database in the SQL world, mapping is similar to the create table definition.

Elasticsearch can understand the structure of the document that you are indexing (reflection) and create the mapping definition automatically. This is called explicit mapping creation.

Getting ready

To execute the code in this recipe, you will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, you can use any HTTP client, such as curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar platforms. I suggest using the Kibana console to provide code completion and better character escaping for Elasticsearch.

To understand the examples and code in this recipe, basic knowledge of JSON is required.

How to do it…

You can explicitly create a mapping by adding a new document to Elasticsearch. For this, perform the following steps:

1. Create an index, as shown in the following code:

   PUT test

The output will be as follows:

{ "acknowledged" : true, "shards_acknowledged" : true,
 "index" : "test" }

2. Put a document in the index, as shown in the following code:

   PUT test/_doc/1
   {"name":"Paul", "age":35}

The output will be as follows:

{
  "_index" : "test", "_id" : "1", "_version" : 1,
  "result" : "created",
  "_shards" : {"total" : 2, "successful" : 1, "failed" : 0 },
  "_seq_no" : 0,  "_primary_term" : 1
}

3. Get the mapping with the following code:

   GET test/_mapping

4. The mapping that's auto-created by Elasticsearch should look as follows:

   {
     "test" : {
       "mappings" : {
         "properties" : {
           "age" : { "type" : "long" },
           "name" : {
             "type" : "text",
             "fields" : {
               "keyword" : {"type" : "keyword", "ignore_above" : 256 }
   } } } } } }

5. To delete the index, you can use the following command:

   DELETE test

The output will be as follows:

{ "acknowledged" : true }

How it works…

The first command line (Step 1) creates an index where we can configure the mappings in the future, if required, and store documents in it.

The second command (Step 2) inserts a document in the index (we'll learn how to create the index in the Creating an index recipe of Chapter 3, Basic Operations, and record indexing in the Indexing a document recipe of Chapter 3, Basic Operations).

Elasticsearch reads all the default properties for the field of the mapping and starts to process them as follows:

• If the field is already present in the mapping and the value of the field is valid (it matches the correct type), Elasticsearch does not need to change the current mappings.
• If the field is already present in the mapping but the value of the field is of a different type, it tries to upgrade the field type (that is, from integer to long). If the types are not compatible, it throws an exception, and the indexing process fails.
• If the field is not present, it tries to auto-detect the type of field. It updates the mappings with a new field mapping. (In the case of a null value, it skips the mapping update until it encounters a concrete type.)

There's more…

In Elasticsearch, every document has a unique identifier, called an ID for a single index, which is stored in the special _id field of the document.

The _id field can be provided at index time or can be assigned automatically by Elasticsearch if it is missing.

When a mapping type is created or changed, Elasticsearch automatically propagates mapping changes to all the nodes in the cluster so that all the shards are aligned to process that particular type.

In Elasticsearch 7.x, there was a default type (_doc): it was removed in Elasticsearch 8.x and above.

See also

Please refer to the following recipes in Chapter 3, Basic Operations:

• The Creating an index recipe, which is about putting new mappings in an index while it's being created
• The Putting a mapping in an index recipe, which is about extending a mapping in an index

Using explicit mapping makes it possible to start to quickly ingest the data using a schemaless approach without being concerned about field types. Thus, to achieve better results and performance in indexing, it's required to manually define a mapping.

Fine-tuning mapping brings some advantages, such as the following:

• Reducing the index size on the disk (disabling functionalities for custom fields)
• Indexing only interesting fields (general speed up)
• Precooking data for fast search or real-time analytics (such as aggregations)
• Correctly defining whether a field must be analyzed in multiple tokens or considered as a single token
• Defining mapping types such as geo point, suggester, vectors, and so on

Elasticsearch allows you to use base fields with a wide range of configurations.

Getting ready

You will need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started. 

To execute the commands in this recipe, you can use any HTTP client, such as curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console, which provides code completion and better character escaping for Elasticsearch.

To execute this recipe's examples, you will need to create an index with a test name, where you can put mappings, as explained in the Using explicit mapping creation recipe.

How to do it...

Let's use a semi real-world example of a shop order for our eBay-like shop:

1. First, we must define an order:

Figure 2.1 – Example of an order

2. Our order record must be converted into an Elasticsearch mapping definition, as follows:

   PUT test/_mapping
   {  "properties" : {
         "id" : {"type" : "keyword"},
         "date" : {"type" : "date"},
         "customer_id" : {"type" : "keyword"},
         "sent" : {"type" : "boolean"},
         "name" : {"type" : "keyword"},
         "quantity" : {"type" : "integer"},
         "price" : {"type" : "double"},
         "vat" : {"type" : "double", "index": false}
   } }

Now, the mapping is ready to be put in the index. We will learn how to do this in the Putting a mapping in an index recipe of Chapter 3, Basic Operations.

How it works...

Field types must be mapped to one of the Elasticsearch base types, and options on how the field must be indexed need to be added.

The following table is a reference for the mapping types:

Figure 2.2 – Base type mapping

Depending on the data type, it's possible to give explicit directives to Elasticsearch when you're processing the field for better management. The most used options are as follows:

• store (default false): This marks the field to be stored in a separate index fragment for fast retrieval. Storing a field consumes disk space but reduces computation if you need to extract it from a document (that is, in scripting and aggregations). The possible values for this option are true and false. They are always retuned as an array of values for consistency.

The stored fields are faster than others in aggregations.

• index: This defines whether or not the field should be indexed. The possible values for this parameter are true and false. Index fields are not searchable (the default is true).
• null_value: This defines a default value if the field is null.
• boost: This is used to change the importance of a field (the default is 1.0).

boost works on a term level only, so it's mainly used in term, terms, and match queries.

• search_analyzer: This defines an analyzer to be used during the search. If it's not defined, the analyzer of the parent object is used (the default is null).
• analyzer: This sets the default analyzer to be used (the default is null).
• norms: This controls the Lucene norms. This parameter is used to score queries better. If the field is only used for filtering, it's a best practice to disable it to reduce resource usage (true for analyzed fields and false for not_analyzed ones).
• copy_to: This allows you to copy the content of a field to another one to achieve functionalities, similar to the _all field.
• ignore_above: This allows you to skip the indexing string if it's bigger than its value. This is useful for processing fields for exact filtering, aggregations, and sorting. It also prevents a single term token from becoming too big and prevents errors due to the Lucene term's byte-length limit of 32,766. The maximum suggested value is 8191 (https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html).

There's more...

From Elasticsearch version 6.x onward, as shown in the Using explicit mapping creation recipe, the explicit inferred type for a string is a multifield mapping:

• The default processing is text. This mapping allows textual queries (that is, term, match, and span queries). In the example provided in the Using explicit mapping creation recipe, this was name.
• The keyword subfield is used for keyword mapping. This field can be used for exact term matching and aggregation and sorting. In the example provided in the Using explicit mapping creation recipe, the referred field was name.keyword.

Another important parameter, available only for text mapping, is term_vector (the vector of terms that compose a string). Please refer to the Lucene documentation for further details at https://lucene.apache.org/core/8_7_0/core/org/apache/lucene/index/Terms.html.

term_vector can accept the following values:

• no: This is the default value; that is, skip term vector.
• yes: This is the store term vector.
• with_offsets: This is the store term vector with a token offset (start, end position in a block of characters).
• with_positions: This is used to store the position of the token in the term vector.
• with_positions_offsets: This stores all the term vector data.
• with_positions_payloads: This is used to store the position and payloads of the token in the term vector.
• with_positions_offsets_payloads: This stores all the term vector data with payloads.

Term vectors allow fast highlighting but consume disk space due to storing additional text information. It's a best practice to only activate it in fields that require highlighting, such as title or document content.

See also

You can refer to the following sources for further details on the concepts of this chapter:

• The online documentation on Elasticsearch provides a full description of all the properties for the different mapping fields at https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-params.html.
• The Specifying a different analyzer recipe at the end of this chapter shows alternative analyzers to the standard one.
• For newcomers who want to explore the concepts of tokenization, I would suggest reading the official Elasticsearch documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenizers.html.

Array or multi-value fields are very common in data models (such as multiple phone numbers, addresses, names, aliases, and so on), but they're not natively supported in traditional SQL solutions.

In SQL, multi-value fields require you to create accessory tables that must be joined to gather all the values, leading to poor performance when the cardinality of the records is huge.

Elasticsearch, which works natively in JSON, provides support for multi-value fields transparently.

Getting ready

You will need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute the commands in this recipe, you can use any HTTP client, such as curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console, which provides code completion and better character escaping for Elasticsearch.

How to do it…

To use an Array type in our mapping, perform the following steps:

1. Every field is automatically managed as an array. For example, to store tags for a document, the mapping would be as follows:

   {  "properties" : {
         "name" : {"type" : "keyword"},
         "tag" : {"type" : "keyword", "store" : true},
         ...
   }

2. This mapping is valid for indexing both documents. The following is the code for document1:

   {"name": "document1", "tag": "awesome"}

3. The following is the code for document2:

   {"name": "document2", "tag": ["cool", "awesome", "amazing"] }

How it works…

Elasticsearch transparently manages the array: there is no difference if you declare a single value or a multi-value due to its Lucene core nature.

Multi-values for fields are managed in Lucene, so you can add them to a document with the same field name. For people with a SQL background, this behavior may be quite strange, but this is a key point in the NoSQL world as it reduces the need for a join query and creates different tables to manage multi-values. An array of embedded objects has the same behavior as simple fields.