The split-brain syndrome

Segal's law states that "A man with a watch knows what time it is. A man with two watches is never sure." Similar considerations apply to databases. It is extremely important to avoid situations where more than one database instance could be accepting writes from your application. Your data will end up in two places, and you will not be able to merge the two databases back together without a large data recovery effort. In effect, this causes you to lose data, so you fail on your durability guarantees. This situation is called a split-brain, and all cluster management systems must have ways to avoid it. The tools used to avoid split-brain are called quorum and fencing.

To understand quorum, consider what happens if the network breaks in a way that applications can see all the cluster members but the cluster members are split into two groups that can't communicate with each other. The nodes on each side must understand what has happened to correctly handle the situation...