Cassandra cluster – replacing a dead node

This section captures the various situations and scenarios that can occur and cause failures in a Cassandra cluster. We will also equip you with the knowledge and talk about the steps to handle these situations. These situations are specific to version 1.1.6 but can be applied to others as well.

Say, this is the problem: you're running an n node, for example let's say there are three node clusters and from that one node goes down; this will result in unrecoverable hardware failure. The solution is this: replace the dead nodes with new nodes.

The following are the steps to achieve the solution:

1. Confirm the node failure using the nodetool ring command:

   bin/nodetool ring -h hostname
   

2. The dead node will be shown as DOWN; let's assume node3 is down:

   192.168.1.54 datacenter1rack1 Up  Normal 755.25 MB 50.00% 0
   192.168.1.55 datacenter1rack1 Down Normal 400.62 MB 25.00%  42535295865117307932921825928971026432
   192.168.1.56 datacenter1rack1 Up  Normal 793.06 MB...