Extending observability past Prometheus
Setting aside the obvious nuance that “no system is truly and fully observable,” let’s focus on how we can get as close as possible to a fully observable system. Observability is all about the concept of being able to account for “unknown unknowns.” In other words, you shouldn’t need to know in advance the various ways that a system can break in order to monitor and observe it effectively.
Knowing what you don’t know
The concept of “unknown unknowns” was popularized by former United States Secretary of Defense, Donald Rumsfeld. It has to do with the idea of not knowing what you don’t know or your ignorance of the extent of your ignorance. In contrast, a “known unknown” would be something that you are aware but do not know. When relating this concept to Prometheus, an “unknown unknown” would be a metric that doesn’t exist, but a known...
