Alerting

Alerting is super important for critical systems. You can plan and build resiliency features as much as you want, but you will never build a failproof system. The right mindset for building robust and reliable systems is to try to minimize failures, but also acknowledge that failures will happen. When failures do happen, you need quick detection and have to alert the right people so that they can investigate and address the problem. Note that I said explicitly alerting people. If your system has self-healing capabilities, then you may be interested in viewing a report of the issues that the system was able to rectify itself. I don't consider those failures, because the system is designed to handle them. For example, containers can crash as much as they want; the kubelet will keep restarting them. A container crash is not considered a failure from a Kubernetes point...