In order to avoid split-brain, Patroni should ensure that the PostgreSQL primary/master node does not accept any writes when the leader key of the node is expired in the Distributed Consensus Store (DCS). For our use case, DCS is etcd. This can be done by letting Patroni shut down PostgreSQL. But what if Patroni itself is hung or killed? In this case, the PostgreSQL node can still take writes. This could cause a split-brain problem, where there could be multiple nodes accepting writes from the application.
To avoid such exceptions that could cause a split-brain, Patroni uses Watchdog. A software watchdog, also known as a softdog, should receive a timer reset request before making a decision about a node restart. Patroni activates the watchdog timer before starting any PostgreSQL node as a primary node. If watchdog activation cannot happen, then Patroni will not be able to start the PostgreSQL node as a primary, nor would it attempt to promote...
