Hadoop Metrics

Most of the Hadoop components reveal the status of their internal components via a metrics subsystem. The idea is to keep counters specific to a given Hadoop process and redirect them according to the configuration to a proper consumer.

The Hadoop Metrics subsystem has several versions. The older one is called metrics1 (or just metrics), and the newer one is referred to as metrics2. Metrics2 is available starting CHD4 and we will focus on this version.

Metrics2 has a notion of sources, sinks, and contexts. A source is any component that records internal statistics, such as NameNode or JobTracker. Sources collect metrics in various contexts. For example, NameNode can reveal information about the JVM it is running in via jvm context, information about the HDFS state via dfs context, and information about RPC via rpc context. Sink is a consumer for metrics. A sink can be a text file, a file for a specific monitoring system, and so on.

By default, Hadoop components collect metrics...