Using an outage to test availability

Every high availability cluster must possess the capability to detect and route around server failures. Hardware faults, virtual instance crashes, mistyped commands, and any number of potential disasters lurk around every corner. The best way to determine the true resilience of our stack is to test it by breaking something.

Let's see what happens by attacking Patroni directly.

Getting ready

This recipe depends on the presence of the entire stack. Please complete all recipes until Installing and configuring HAProxy before continuing.

How to do it...

For this recipe, we should already have three PostgreSQL servers. As usual, we'll assume they are named pg1, pg2, and pg3. If pg2 is the current primary, follow these steps to simulate a server failure:

1. Execute the following command as the postgres user on pg2:

pkill -f patroni

2. Follow the Patroni log on pg1 or pg3 with this command:

tail -f /var/log/postgresql/patroni.log

How it works...

This recipe relies on a dirty...