Elasticsearch runs on Linux/macOS X/Windows and its only requirement is to have Java 8.x installed. Usually, I recommend using the Oracle JDK, which is available at https://github.com/aparo/elasticsearch-7.x-cookbook.

Elasticsearch has an active community and the release cycles are very fast.

Because Elasticsearch depends on many common Java libraries (Lucene, Guice, and Jackson are the most famous ones), the Elasticsearch community tries to keep them updated and fixes bugs that are discovered in them and in the Elasticsearch core. The large user base is also a source of new ideas and features for improving Elasticsearch use cases.

For these reasons, if possible, it's best to use the latest available release (usually the more stable and bug-free one).

To install Elasticsearch, you need a supported operating system (Linux/macOS X/Windows) with a Java Java virtual machine (JVM) 1.8 or higher installed (the Sun Oracle JDK is preferred. More information on this can be found at http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html). A web browser is required to download the Elasticsearch binary release. At least 1 GB of free disk space is required to install Elasticsearch.

We will start by downloading Elasticsearch from the web. The latest version is always downloadable at https://www.elastic.co/downloads/elasticsearch. The versions that are available for different operating systems are as follows:

• elasticsearch-{version-number}.zip and elasticsearch-{version-number}.msi are for the Windows operating systems.
• elasticsearch-{version-number}.tar.gz is for Linux/macOS X, while elasticsearch-{version-number}.deb is for Debian-based Linux distributions (this also covers the Ubuntu family); this is installable with Debian using the dpkg -i elasticsearch-*.deb command.
• elasticsearch-{version-number}.rpm is for Red Hat-based Linux distributions (this also covers the Cent OS family). This is installable with the rpm -i elasticsearch-*.rpm command.

Extract the binary content. After downloading the correct release for your platform, the installation involves expanding the archive in a working directory.

For the Windows platform, a good directory in which to install Elasticsearch could be c:\es, on Unix and /opt/es on macOS X.

To run Elasticsearch, you need a JVM 1.8 or higher installed. For better performance, I suggest that you use the latest Sun/Oracle version.

Let's start Elasticsearch to check if everything is working. To start your Elasticsearch server, just access the directory, and for Linux and macOS X execute the following:

# bin/elasticsearch

Alternatively, you can type the following command line for Windows:

# bin\elasticserch.bat

Your server should now start up and show logs similar to the following:

[2018-10-28T16:19:41,189][INFO ][o.e.n.Node ] [] initializing ...
 [2018-10-28T16:19:41,245][INFO ][o.e.e.NodeEnvironment ] [fyBySLM] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [141.9gb], net total_space [465.6gb], types [apfs]
 [2018-10-28T16:19:41,246][INFO ][o.e.e.NodeEnvironment ] [fyBySLM] heap size [989.8mb], compressed ordinary object pointers [true]
 [2018-10-28T16:19:41,247][INFO ][o.e.n.Node ] [fyBySLM] node name derived from node ID [fyBySLMcR3uqKiYC32P5Sg]; set [node.name] to override
 [2018-10-28T16:19:41,247][INFO ][o.e.n.Node ] [fyBySLM] version[6.4.2], pid[50238], build[default/tar/04711c2/2018-09-26T13:34:09.098244Z], OS[Mac OS X/10.14/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
 [2018-10-28T16:19:41,247][INFO ][o.e.n.Node ] [fyBySLM] JVM arguments [-Xms1g, -Xmx1g,
... truncated ...
 [2018-10-28T16:19:42,511][INFO ][o.e.p.PluginsService ] [fyBySLM] loaded module [aggs-matrix-stats]
 [2018-10-28T16:19:42,511][INFO ][o.e.p.PluginsService ] [fyBySLM] loaded module [analysis-common]
 ...truncated...
[2018-10-28T16:19:42,513][INFO ][o.e.p.PluginsService ] [fyBySLM] no plugins loaded
 ...truncated...
[2018-10-28T16:19:46,776][INFO ][o.e.n.Node ] [fyBySLM] initialized
 [2018-10-28T16:19:46,777][INFO ][o.e.n.Node ] [fyBySLM] starting ...
 [2018-10-28T16:19:46,930][INFO ][o.e.t.TransportService ] [fyBySLM] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
 [2018-10-28T16:19:49,983][INFO ][o.e.c.s.MasterService ] [fyBySLM] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {fyBySLM}{fyBySLMcR3uqKiYC32P5Sg}{-pUWNdRlTwKuhv89iQ6psg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
 ...truncated...
[2018-10-28T16:19:50,452][INFO ][o.e.l.LicenseService ] [fyBySLM] license [b2754b17-a4ec-47e4-9175-4b2e0d714a45] mode [basic] - valid

The Elasticsearch package generally contains the following directories:

• bin: This contains the scripts to start and manage Elasticsearch.
• elasticsearch.bat: This is the main executable script to start Elasticsearch.
• elasticsearch-plugin.bat: This is a script to manage plugins.
• config: This contains the Elasticsearch configs. The most important ones are as follows:
  • elasticsearch.yml: This is the main config file for Elasticsearch
  • log4j2.properties: This is the logging config file
• lib: This contains all the libraries required to run Elasticsearch.
• logs: This directory is empty at installation time, but in the future, it will contain the application logs.
• modules: This contains the Elasticsearch default plugin modules.
• plugins: This directory is empty at installation time, but it's the place where custom plugins will be installed.

During Elasticsearch startup, the following events happen:

• A node name is generated automatically (that is, fyBySLM) if it is not provided in elasticsearch.yml. The name is randomly generated, so it's a good idea to set it to a meaningful and memorable name instead.
• A node name hash is generated for this node, for example, fyBySLMcR3uqKiYC32P5Sg.
• The default installed modules are loaded. The most important ones are as follows:
  • aggs-matrix-stats: This provides support for aggregation matrix stats.
  • analysis-common: This is a common analyzer for Elasticsearch, which extends the language processing capabilities of Elasticsearch.
  • ingest-common: These include common functionalities for the ingest module.
  • lang-expression/lang-mustache/lang-painless: These are the default supported scripting languages of Elasticsearch. 
  • mapper-extras: This provides an extra mapper type to be used, such as token_count and scaled_float.
  • parent-join: This provides an extra query, such as has_children and has_parent.
  • percolator: This provides percolator capabilities.
  • rank-eval: This provides support for the experimental rank evaluation APIs. These are used to evaluate hit scoring based on queries.
  • reindex: This provides support for reindex actions (reindex/update by query).
  • x-pack-*: All the xpack modules depend on a subscription for their activation.
• If there are plugins, they are loaded.
• If not configured, Elasticsearch binds the following two ports on the localhost 127.0.0.1 automatically:
  • 9300: This port is used for internal intranode communication.
  • 9200: This port is used for the HTTP REST API.
• After starting, if indices are available, they are restored and ready to be used.

If these port numbers are already bound, Elasticsearch automatically increments the port number and tries to bind on them until a port is available (that is, 9201, 9202, and so on).

There are more events that are fired during Elasticsearch startup. We'll see them in detail in other recipes.

During a node's startup, a lot of required services are automatically started. The most important ones are as follows:

• Cluster services: This helps you manage the cluster state and intranode communication and synchronization
• Indexing service: This helps you manage all the index operations, initializing all active indices and shards
• Mapping service: This helps you manage the document types stored in the cluster (we'll discuss mapping in Chapter 2, Managing Mapping)
• Network services: This includes services such as HTTP REST services (default on port 9200), and internal Elasticsearch protocol (port 9300) if the thrift plugin is installed
• Plugin service: This manages loading the plugin 
• Aggregation services: This provides advanced analytics on stored Elasticsearch documents such as statistics, histograms, and document grouping
• Ingesting services: This provides support for document preprocessing before ingestion such as field enrichment, NLP processing, types conversion, and automatic field population
• Language scripting services: This allows you to add new language scripting support to Elasticsearch

The Setting up networking recipe we're going to cover next will help you with the initial network setup. Check the official Elasticsearch download page at https://www.elastic.co/downloads/elasticsearch to get the latest version.

Correctly setting up networking is very important for your nodes and cluster.

There are a lot of different installation scenarios and networking issues. The first step for configuring the nodes to build a cluster is to correctly set the node discovery.

To change configuration files, you will need a working Elasticsearch installation and a simple text editor, as well as your current networking configuration (your IP).

To setup the networking, use the following steps:

1. Using a standard Elasticsearch configuration config/elasticsearch.yml file, your node will be configured to bind on the localhost interface (by default) so that it can't be accessed by external machines or nodes.
2. To allow another machine to connect to our node, we need to set network.host to our IP (for example, I have 192.168.1.164).
3. To be able to discover other nodes, we need to list them in the discovery.zen.ping.unicast.hosts parameter. This means that it sends signals to the machine in a unicast list and waits for a response. If a node responds to it, they can join in a cluster.

4. In general, from Elasticsearch version 6.x, the node versions are compatible. You must have the same cluster name (the cluster.name option in elasticsearch.yml) to let nodes join with each other.

5. To customize the network preferences, you need to change some parameters in the elasticsearch.yml file, as follows:

cluster.name: ESCookBook
node.name: "Node1"
network.host: 192.168.1.164
discovery.zen.ping.unicast.hosts: ["192.168.1.164","192.168.1.165[9300-9400]"]

6. This configuration sets the cluster name to Elasticsearch, the node name, the network address, and it tries to bind the node to the address given in the discovery section by performing the following tasks:

• • We can check the configuration during node loading
  • We can now start the server and check whether the networking is configured, as follows:

[2018-10-28T17:42:16,386][INFO ][o.e.c.s.MasterService ] [Node1] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {Node1}{fyBySLMcR3uqKiYC32P5Sg}{IX1wpA01QSKkruZeSRPlFg}{192.168.1.164}{192.168.1.164:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
 [2018-10-28T17:42:16,390][INFO ][o.e.c.s.ClusterApplierService] [Node1] new_master {Node1}{fyBySLMcR3uqKiYC32P5Sg}{IX1wpA01QSKkruZeSRPlFg}{192.168.1.164}{192.168.1.164:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {Node1}{fyBySLMcR3uqKiYC32P5Sg}{IX1wpA01QSKkruZeSRPlFg}{192.168.1.164}{192.168.1.164:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
 [2018-10-28T17:42:16,403][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [Node1] publish_address {192.168.1.164:9200}, bound_addresses {192.168.1.164:9200}
 [2018-10-28T17:42:16,403][INFO ][o.e.n.Node ] [Node1] started
 [2018-10-28T17:42:16,600][INFO ][o.e.l.LicenseService ] [Node1] license [b2754b17-a4ec-47e4-9175-4b2e0d714a45] mode [basic] - valid

As you can see from my screen dump, the transport is bound to 192.168.1.164:9300. The REST HTTP interface is bound to 192.168.1.164:9200.

The following are the main important configuration keys for networking management:

• cluster.name: This sets up the name of the cluster. Only nodes with the same name can join together.
• node.name: If not defined, this is automatically assigned by Elasticsearch.

node.name allows defining a name for the node. If you have a lot of nodes on different machines, it is useful to set their names to something meaningful in order to easily locate them. Using a valid name is easier to remember than a generated name such as fyBySLMcR3uqKiYC32P5Sg.

network.host defines the IP of your machine to be used to bind the node. If your server is on different LANs, or you want to limit the bind on only one LAN, you must set this value with your server IP.

discovery.zen.ping.unicast.hosts allows you to define a list of hosts (with ports or a port range) to be used to discover other nodes to join the cluster. The preferred port is the transport one, usually 9300.

The addresses of the hosts list can be a mix of the following:

• Hostname, that is, myhost1
• IP address, that is, 192.168.1.12
• IP address or hostname with the port, that is, myhost1:9300, 192.168.168.1.2:9300
• IP address or hostname with a range of ports, that is, myhost1:[9300-9400], 192.168.168.1.2:[9300-9400]

The Setting up a node recipe in this chapter

Elasticsearch allows the customization of several parameters in an installation. In this recipe, we'll see the most used ones to define where to store our data and improve overall performance.

As described in the downloading and installing Elasticsearch recipe, you need a working Elasticsearch installation and a simple text editor to change configuration files.

The steps required for setting up a simple node are as follows:

1. Open the config/elasticsearch.yml file with an editor of your choice.
2. Set up the directories that store your server data, as follows:

• For Linux or macOS X, add the following path entries (using /opt/data as the base path):

path.conf: /opt/data/es/conf
path.data: /opt/data/es/data1,/opt2/data/data2
path.work: /opt/data/work
path.logs: /opt/data/logs
path.plugins: /opt/data/plugins

• For Windows, add the following path entries (using c:\Elasticsearch as the base path):

path.conf: c:\Elasticsearch\conf
path.data: c:\Elasticsearch\data
path.work: c:\Elasticsearch\work
path.logs: c:\Elasticsearch\logs
path.plugins: c:\Elasticsearch\plugins

3. Set up the parameters to control the standard index shard and replication at creation. These parameters are as follows:

index.number_of_shards: 1
index.number_of_replicas: 1

The path.conf parameter defines the directory that contains your configurations, mainly elasticsearch.yml and logging.yml. The default is $ES_HOME/config, with ES_HOME to install the directory of your Elasticsearch server.

The path.data parameter is the most important one. This allows us to define one or more directories (in a different disk) where you can store your index data. When you define more than one directory, they are managed similarly to RAID 0 (their space is sum up), favoring locations with the most free space.

The path.work parameter is a location in which Elasticsearch stores temporary files.

The path.log parameter is where log files are put. These control how a log is managed in logging.yml.

The path.plugins parameter allows you to override the plugins path (the default is $ES_HOME/plugins). It's useful to put system-wide plugins in a shared path (usually using NFS) in case you want a single place where you store your plugins for all of the clusters.

The main parameters are used to control index and shards in index.number_of_shards, which controls the standard number of shards for a new created index, and index.number_of_replicas, which controls the initial number of replicas.

Refer to the following points to learn more about topics related to this recipe:

• The Setting up for Linux systems recipe
• You can refer to the official Elasticsearch documentation at https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html.