Understanding cluster, replication, and sharding
Related to shard management, there is the key concept of replication and cluster status.
Getting ready
You need one or more nodes running to have a cluster. To test an effective cluster you need at least two nodes (they can be on the same machine).
How it works...
An index can have one or more replicas—the shards are called primary if they are part of the master index and secondary if they are part of replicas.
To maintain consistency in write operations the following workflow is executed:
- The write is first executed in the primary shard.
- If the primary write is successfully done, it is propagated simultaneously in all the secondary shards.
- If a primary shard dies, a secondary one is elected as primary (if available) and the flow is re-executed.
During search operations, a valid set of shards is chosen randomly between primary and secondary to improve performances.
The following figure shows an example of possible shards configuration:
Best practice
In order to prevent data loss and to have High Availability, it's good to have at least one replica so that your system can survive a node failure without downtime and without loss of data.
There's more…
Related to the concept of replication there is the cluster indicator of the health of your cluster.
It can cover three different states:
- Green: Everything is ok.
- Yellow: Something is missing but you can work.
- Red: "Houston, we have a problem". Some primary shards are missing.
How to solve the yellow status
Mainly yellow status is due to some shards that are not allocated. If your cluster is in recovery status, just wait if there is enough space in nodes for your shards.
If your cluster, even after recovery is still in yellow state, it means you don't have enough nodes to contain your replicas so you can either reduce the number of your replicas or add the required number of nodes.
Best practice
The total number of nodes must not be lower than the maximum number of replicas.
How to solve the red status
When you have lost data (that is, one or more shard is missing), you need to try restoring the node(s) that are missing. If your nodes restart and the system goes back to yellow or green status you are safe. Otherwise, you have lost data and your cluster is not usable. In this case, delete the index/indices and restore them from backup (if you have it) or from other sources.
Best practice
To prevent data loss, I suggest having always at least two nodes and the replica set to 1.
Tip
Having one or more replicas on different nodes on different machines allows you to have a live backup of your data, always updated.
See also
- Replica and shard management in this chapter.
