Monitoring cluster performance

Since services such as Synapse and ADF are platform-as-a-service (PaaS) services where you will not have explicit control over the clusters, the one place where we can control each and every aspect of a cluster is the Azure HDInsight service. In HDInsight, you can create your own Hadoop, Spark, Hive, HBase, and other clusters and control every aspect of the cluster. You can use Log Analytics to monitor cluster performance, as with the other examples we saw earlier in the chapter. You can learn more about using Log Analytics in HDInsight here: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-oms-log-analytics-tutorial.

Apart from the Log Analytics approach, there are four main areas of the HDInsight portal that help monitor cluster performance. Let's look at them.

Monitoring overall cluster performance

The HDInsight Ambari dashboard is the first place to check for cluster health. If you see very high heap usage, disk usage...