Connecting the dots across observability systems
Prometheus is quickly becoming many people’s first introduction to the world of observability technologies. Its popularity and seeming ubiquity have consequently led it to influence many other observability projects that seek to replicate its ease of use and simple data model. This has the added benefit of making it easier to jump between observability services without having to change the filters we use to select data.
The most meaningful way to easily connect the dots across our various observability systems is to ensure consistency in the metadata that is attached to the telemetry coming from that system. In the context of Prometheus, that means that our label keys and values for ServiceX should be the same in Prometheus as they are in the rest of the systems we use. For example, if your Prometheus metrics for ServiceX have a label app=servicex on them, your logs for ServiceX should not have a label of app=service-x on them...
