Summary
Understanding the different options for sampling provides us with the ability to manage the amount of data produced by our applications. Knowing the trade-offs of different sampling strategies and some of the methods available helps decrease the level of noise in a busy environment.
The OpenTelemetry configuration and samplers available to configure sampling at the application level can help reduce the load and cost upfront in systems via head sampling. Configuring tail sampling at collection time provides the added benefit of making a more informed decision on what to keep or discard. This benefit comes at the added cost of having to run a collection point with sufficient resources to buffer the data until a decision can be reached.
Ultimately, the decisions made when configuring sampling will impact what data is available to observe what is happening in a system. Sample too little and you may miss important events. Sample too much and the cost of producing telemetry...
