Exploring CloudWatch Application Insights

CloudWatch Application Insights streamlines the monitoring of enterprise applications with its intuitive and automated setup process. This reduces the time and effort required to configure monitoring, as it automatically sets up metrics, telemetry, logs, and alarms. The key advantage of using Application Insights is that it automatically discovers and configures application-specific monitoring, making it easy to effectively monitor your applications. Furthermore, it leverages ML analysis to perform in-depth problem analysis based on the gathered data, allowing you to quickly identify correlations between issues and relevant events.

It provides a dashboard for detected problems and provides insights and observations. It has predefined, customizable rules for alerting. It also helps you create an AWS System Manager (SSM) OpsItem to take remediation action with SSM runbooks.

Supported data sources on the Windows operating system include the following:

• Platform Metrics: Built-in platform metrics, advanced custom metrics, CloudWatch metrics
• Application Data: Microsoft IIS Logs, Microsoft SQL Server error log, custom .NET applications, .NET Core, Prometheus Java metrics
• Windows Performance Counters: .NET CLR, W3SVC_W3WP, Interop CLR, SQL server metrics, IIS counters
• Windows Event Logs: System, security, and application
• Extensibility for custom log patterns and custom application tiers

CloudWatch Application Insights supports gathering metrics, logs, and application-specific metrics using the console. Currently, X-Ray tracing is not supported.

You can use two different methods to discover the application using Application Insights:

• Resource group based: You can create a new resource group for an application using AWS resource group and tags as a component of your CloudFormation template. As part of the exercise in this section, we will use this method to create a new resource group.
• Account based: By implementing the account-based method, you can create application monitoring for all the resources in the AWS account. This is ideal when you are looking to instrument all future applications deployed in your AWS account.

In the previous chapter, we deployed a Java application on an EC2 instance and set up observability as a part of CloudFormation. We will set up observability for the same application using Application Insights and get additional insights related to the application besides the path of the user journey provided by X-Ray.

You can download the changed template for this exercise. To do so, log into your AWS account and click or copy and paste into your browser window the following Quickstart URL:

https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=scorekeepappinsight&templateURL=https://insiders-guide-observability-on-aws-book.s3.amazonaws.com/chapter-05/basic-ec2-template_rg_nomonitoring.yml

If you would like to download the template and verify it before deploying, you can download it from the following URL:

https://insiders-guide-observability-on-aws-book.s3.amazonaws.com/chapter-05/basic-ec2-template_rg_nomonitoring.yml

This action will open the AWS CloudFormation console with all the parameters with default values. You can keep the default values and click on the Next button on all the subsequent pages. On the last one, named Review scorekeepappinsight, do not forget to check the box with the text The following resource(s) require capabilities: [AWS::IAM::Role] and click on the Create Stack button.

What we have changed in the CloudFormation template is the following:

• The addition of YAML to create a resource group
• Modification of the IAM role to allow management using AWS Systems Manager (as Application Insights leverages AWS SSM to install and configure the CloudWatch agent)
• The removal of the installation and configuration of the CloudWatch agent on the EC2 instance

Let’s deploy the application using CloudFormation and provide the stack name to start with: scorekeep. Once it’s successfully deployed, let’s discover the application using CloudWatch Application Insights:

1. Navigate to Application Insights | Add an application | Resource group based application | Confirm.

Figure 5.30 – Resource group based discovery in Application Insights

2. From the dropdown, select my_resource_group and select Automatic monitoring of new resources, Monitor EventBridge events, and Integrate with AWS Systems Manager OpsCenter and click Next.

Figure 5.31 – Discovery of resources using resource groups

3. CloudWatch Application Insights discovers all the resources that are deployed as a part of the CloudFormation template. You can add the components installed on EC2 from the dropdown as shown in the following screenshot, which will add additional applications to CloudWatch monitoring.

Figure 5.32 – Application components discovered by Application Insights

4. In Specify component details, you can also add additional logs to be gathered from the Java application leveraging the CloudWatch agent. Add the log file /tmp/scorekeep.log. We can also change it at a later stage to accommodate additional run operations if necessary.

Figure 5.33 – Adding logs to CloudWatch monitoring

5. Select Next and Submit to start the onboarding of the application.

Figure 5.34 – Submit application monitoring

6. You can verify that the resource group has been added successfully using Application Insights.

Figure 5.35 – Successfully added application

You can see that it is additionally asking to set up X-Ray, which is already done as a part of the CloudFormation template and application instrumentation, as discussed in Chapter 4, Implementing Distributed Tracing Using AWS X-Ray.

7. Once you navigate to Application Insights, you will see monitored assets, telemetry, and a summary along with the detected problems as a unified dashboard.

Figure 5.36 – Detected problems

Application Insights uses CloudFormation to deploy the required resources to monitor EC2 instances, as shown in the following figure. This will also address the challenge of agent installation in the newly built instances as a part of the resource group when you are leveraging Auto Scaling mechanisms as discovery is a continuous process.

Figure 5.37 – CloudFormation template created by Application Insights

8. When you navigate to any problem summary generated by Application Insights, it will provide you with an insights dashboard for the issue and also request feedback. In the following screenshot, I have navigated to the CPU issue, where the RDS database instance has high latency and the proposed action is to scale up the RDS instance.

Figure 5.38 – Problem navigation in Application Insights

9. As we also created an OpsItem in AWS Systems Manager, it has provided a summary of the related resources and provided navigation of CloudTrail (audit events), CloudFormation (resources), and also CloudWatch alarms in a single view.

Figure 5.39 – Related resources view in AWS Systems Manager OpsCenter

Application Insights provides a way to setup collecting infrastucutre, application related metrices and logs and setup alarms and anomaly detection automatically along with installation of CloudWatch agent in in few clicks compared to instrumenting the application manually. We can also implement Application Insights using CloudFormation by using the resource type AWS::ApplicationInsights:Application: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-applicationinsights-application.html.

Now let’s explore how to derive operational intelligence from CloudWatch Logs using CloudWatch Logs Insights.