Adding a node back into the cluster
Recovering systems after a major crash or outage is not an enjoyable experience. We must reboot or restore one or more servers, perform forensics to determine the root cause of the failure, and attempt to repair or replace corrupt binaries.
This is no less true on systems that rely on Patroni as their high availability solution. However, Patroni automates the more annoying portions of recovering a damaged PostgreSQL database.
Let's see how.
Getting ready
This recipe depends on the presence of the entire stack. Please complete all previous recipes in this chapter before continuing.
We also need a broken server. The easiest way to do this is to break it ourselves. Execute these commands on any system to simulate an unrecoverable server crash:
pkill -9 patronipkill -9 postgresfind /db/pgdata -name '*r*' -o -name '*0*' -delete
How to do it...
For this recipe, we should already have three PostgreSQL servers. As before, we'll assume they are named pg1
, pg2
, and pg3...