Understanding alerting concepts
Hopefully, without running the risk of being overly pedantic, a few declarations can be made here about alerting and how certain aspects of alerting (especially with respect to anomaly detection) are extremely important to understand before we get into the mechanics of configuring those alerts.
Anomalies are not necessarily alerts
This needs to be explicitly said. Often, users who first embrace anomaly detection feel compelled to alert on everything once they realize that you can alert on anomalies. This is potentially a really challenging situation if anomaly detection is deployed across hundreds, thousands, or even tens of thousands of entities. Anomaly detection, while certainly liberating users from having to define specific, rule-driven exceptions or hardcoded thresholds from alerts, also has the potential to be deployed broadly across a lot of data. We need to be cognizant that detailed alerting on every little anomaly could be potentially...