Chapter 5. Monitoring Applications
Distributed systems fail often. Worse, they often fail partially. When failures occur during operations responsible for altering a system's state (for instance, a write or delete operation), how can the correct state be recovered, especially when these operations are concurrent? To make matters even worse, some operations fail silently. Partial failures, then, can put applications in an indeterminate state. It is difficult to predict how an opaque system will behave.
Consider this quote from The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines:
"Suppose a cluster has ultra-reliable server nodes with a stellar mean time between failures (MTBF) of 30 years (10,000 days)—well beyond what is typically possible to achieve at a realistic cost. Even with these ideally reliable servers, a cluster of 10,000 servers will see an average of one server failure per day. Thus, any application that needs the...
