Monitoring node metrics
Whether you are using JMX monitoring software or the nodetool
utility, it is important to know what statistics are worth watching. The names and locations of specific attributes can vary depending upon the Cassandra version, but the key ideas remain the same.
The objective here is to give you an understanding of the available statistics so you will know how to choose the proper monitoring targets. We will use nodetool
for this purpose as its options tend to be more stable. You should find it straightforward to locate the equivalent JMX MBean.
Thread pools
Cassandra's design employs a staged event-driven architecture (SEDA) that essentially comprises message queues (containing events) feeding into thread pools (or stages). The stages fire off messages to other stages via a messaging service. There are stages to handle a variety of tasks.
Running nodetool tpstats
provides detailed information about what's happening at each of the stages. A buildup of pending tasks in any...