HAProxy and PgBouncer are an ideal combination for performing high-availability node maintenance in a multi-master context. The reason for this is related to the fact HAProxy uses its detection mechanism to automatically direct traffic to any online node. Since all BDR nodes are writable, this means that any switch from one node to another will allow full application functionality. We can either allow this to happen automatically or we can take direct control.
When we added PgBouncer, this gave us the added capability of transaction management. Before we can stop a PostgreSQL node, it's considered polite to allow any executing transactions to complete their work. By combining the RECONNECT and WAIT_CLOSE operations of PgBouncer with the seamless redirection of HAProxy, we can decommission any BDR node without the application ever knowing...