Designing robust systems
When you want to design a robust system you first need to understand the possible failure modes, the risk/probability of each failure, and the impact/cost of each failure. Then, you can consider various prevention and mitigation measures, loss-cutting strategies, incident-management strategies, and recovery procedures. Finally, you can come up with a plan that matches risks to mitigation profiles, including cost. A comprehensive design is not trivial and needs to be updated as the system evolves. The higher the stakes the more thorough your plan should be. This process has to be tailored for each organization. A corner of error recovery and robustness is detecting failures and being able to troubleshoot. The following sub-sections describe common failure categories, how to detect them, and where to collect additional information.
Hardware failure
Hardware failures in Kubernetes can be divided into two groups:
The node is unresponsive
The node is responsive
When the node...
