Use App Metrics to analyze HTTP traffic, errors & network performance of a .NET Core app [Tutorial]

App Metrics is an open source tool that can be plugged in with the ASP.NET Core applications. It provides real-time insights about how the application is performing and provides a complete overview of the application's health status. It provides metrics in a JSON format and integrates with the Grafana dashboards for visual reporting. App Metrics is based on .NET Standard and runs cross-platform. It provides various extensions and reporting dashboards that can run on Windows and Linux operating system as well. In this article, we will focus on App Metrics, analyse HTTP traffic, errors, and network performance in .NET Core.

This tutorial is an extract from the book C# 7 and .NET Core 2.0 High Performance, authored by Ovais Mehboob Ahmed Khan.

Setting up App Metrics with ASP.NET Core

We can set up App Metrics in the ASP.NET Core application in three easy steps, which are as follows:

1. Install App Metrics.

App Metrics can be installed as NuGet packages. Here are the two packages that can be added through NuGet in your .NET Core project:

      Install-Package App.Metrics 
      Install-Pacakge App.Metrics.AspnetCore.Mvc

2. Add App Metrics in Program.cs.

Add UseMetrics to Program.cs in the BuildWebHost method, as follows:

      public static IWebHost BuildWebHost(string[] args) => 
        WebHost.CreateDefaultBuilder(args) 
          .UseMetrics() 
          .UseStartup<Startup>() 
          .Build();

3. Add App Metrics in Startup.cs.

Finally, we can add a metrics resource filter in the ConfigureServices method of the Startup class as follows:

      public void ConfigureServices(IServiceCollection services) 
      { 
        services.AddMvc(options => options.AddMetricsResourceFilter()); 
      }

4. Run your application.

Build and run the application. We can test whether App Metrics is running well by using URLs, as shown in the following table. Just append the URL to the application's root URL:

Appending /metrics or /metrics-text to the application's root URL gives complete information about application metrics. /metrics returns the JSON response that can be parsed and represented in a view with some custom parsing.

Tracking middleware

With App Metrics, we can manually define the typical web metrics which are essential to record telemetry information. However, for ASP.NET Core, there is a tracking middleware that can be used and configured in the project, which contains some built-in key metrics which are specific to the web application.

Metrics that are recorded by the Tracking middleware are as follows:

• Apdex: This is used to monitor the user's satisfaction based on the overall performance of the application. Apdex is an open industry standard that measures the user's satisfaction based on the application's response time.

We can configure the threshold of time, T, for each request cycle, and the metrics are calculated based on following conditions:

• Response times: This provides the overall throughput of the request being processed by the application and the duration it takes per route within the application.

• Active requests: This provides the list of active requests which have been received on the server in a particular amount of time.

• Errors: This provides the aggregated results of errors in a percentage that includes the overall error request rate, the overall count of each uncaught exception type, the total number of error requests per HTTP status code, and so on.

• POST and PUT sizes: This provides the request sizes for HTTP POST and PUT requests.

Adding tracking middleware

We can add tracking middleware as a NuGet package as follows:

Install-Package App.Metrics.AspNetCore.Tracking

Tracking middleware provides a set of middleware that is added to record telemetry for the specific metric. We can add the following middleware in the Configure method to measure performance metrics:

app.UseMetricsApdexTrackingMiddleware(); 
app.UseMetricsRequestTrackingMiddleware(); 
app.UseMetricsErrorTrackingMiddleware(); 
app.UseMetricsActiveRequestMiddleware(); 
app.UseMetricsPostAndPutSizeTrackingMiddleware(); 
app.UseMetricsOAuth2TrackingMiddleware();

Alternatively, we can also use meta-pack middleware, which adds all the available tracking middleware so that we have information about all the different metrics which are in the preceding code:

app.UseMetricsAllMiddleware();

Next, we will add tracking middleware in our ConfigureServices method as follows:

services.AddMetricsTrackingMiddleware();

In the main Program.cs class, we will modify the BuildWebHost method and add the UseMetricsWebTracking method as follows:

public static IWebHost BuildWebHost(string[] args) => 
  WebHost.CreateDefaultBuilder(args) 
    .UseMetrics() 
    .UseMetricsWebTracking() 
    .UseStartup<Startup>() 
    .Build();

Setting up configuration

Once the middleware is added, we need to set up the default threshold and other configuration values so that reporting can be generated accordingly. The web tracking properties can be configured in the appsettings.json file. Here is the content of the appsettings.json file that contains the MetricWebTrackingOptions JSON key:

"MetricsWebTrackingOptions": { 
  "ApdexTrackingEnabled": true, 
  "ApdexTSeconds": 0.1, 
  "IgnoredHttpStatusCodes": [ 404 ], 
  "IgnoredRoutesRegexPatterns": [], 
  "OAuth2TrackingEnabled": true 
    },

ApdexTrackingEnabled is set to true so that the customer satisfaction report will be generated, and ApdexTSeconds is the threshold that decides whether the request response time was satisfactory, tolerating, or frustrating. IgnoredHttpStatusCodes contains the list of status codes that will be ignored if the response returns a 404 status. IgnoredRoutesRegexPatterns are used to ignore specific URIs that match the regular expression, and OAuth2TrackingEnabled can be set to monitor and record the metrics for each client and provide information specific to the request rate, error rate, and POST and PUT sizes for each client.

Run the application and do some navigation. Appending /metrics-text in your application URL will display the complete report in textual format. Here is the sample snapshot of what textual metrics looks like:

Adding visual reports

There are various extensions and reporting plugins available that provide a visual reporting dashboard. Some of them are GrafanaCloud Hosted Metrics, InfluxDB, Prometheus, ElasticSearch, Graphite, HTTP, Console, and Text File. We will configure the InfluxDB extension and see how visual reporting can be achieved.

Setting up InfluxDB

InfluxDB is the open source time series database developed by Influx Data. It is written in the Go language and is widely used to store time series data for real-time analytics. Grafana is the server that provides reporting dashboards that can be viewed through a browser. InfluxDB can easily be imported as an extension in Grafana to display visual reporting from the InfluxDB database.

Setting up the Windows subsystem for Linux

In this section, we will set up InfluxDB on the Windows subsystem for the Linux operating system.

1. First of all, we need to enable the Windows subsystem for Linux by executing the following command from the PowerShell as an Administrator:

      Enable-WindowsOptionalFeature -Online -FeatureName 
      Microsoft-Windows-Subsystem-Linux

After running the preceding command, restart your computer.

2. Next, we will install Linux distro from the Microsoft store. In our case, we will install Ubuntu from the Microsoft Store. Go to the Microsoft Store, search for Ubuntu, and install it.
3. Once the installation is done, click on Launch:

4. This will open up the console window, which will ask you to create a user account for Linux OS (Operating System).
5. Specify the username and password that will be used.
6. Run the following command to update Ubuntu to the latest stable version from the bash shell. To run bash, open the command prompt, write bash, and hit Enter:

7. Finally, it will ask you to create an Ubuntu username and password. Specify the username and password and hit enter.

Installing InfluxDB

Here, we will go through some steps to install the InfluxDB database in Ubuntu:

1. To set up InfluxDB, open a command prompt in Administrator mode and run the bash shell.
2. Execute the following commands to the InfluxDB data store on your local PC:

      $ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add - 
      $ source /etc/lsb-release 
      $ echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} 
      $ {DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

3. Install InfluxDB by executing the following command:

      $ sudo apt-get update && sudo apt-get install influxdb

4. Execute the following command to run InfluxDB:

      $ sudo influxd

5. Start the InfluxDB shell by running the following command:

      $ sudo influx

It will open up the shell where database-specific commands can be executed.

6. Create a database by executing the following command. Specify a meaningful name for the database. In our case, it is appmetricsdb:

      > create database appmetricsdb 

Installing Grafana

Grafana is an open source tool used to display dashboards in a web interface. There are various dashboards available that can be imported from the Grafana website to display real-time analytics. Grafana can simply be downloaded as a zip file from http://docs.grafana.org/installation/windows/. Once it is downloaded, we can start the Grafana server by clicking on the grafana-server.exe executable from the bin directory.

Grafana provides a website that listens on port 3000. If the Grafana server is running, we can access the site by navigating to http://localhost:3000.

Adding the InfluxDB dashboard

There is an out-of-the-box InfluxDB dashboard available in Grafana which can be imported from the following link: https://grafana.com/dashboards/2125.

Copy the dashboard ID and use this to import it into the Grafana website.

We can import the InfluxDB dashboard by going to the Manage option on the Grafana website, as follows:

From the Manage option, click on the + Dashboard button and hit the New Dashboard option. Clicking on Import Dashboard will lead to Grafana asking you for the dashboard ID:

Paste the dashboard ID (for example, 2125) copied earlier into the box and hit Tab. The system will show the dashboard's details, and clicking on the Import button will import it into the system:

Configuring InfluxDB

We will now configure the InfluxDB dashboard and add a data source that connects to the database that we just created.

To proceed, we will go to the Data Sources section on the Grafana website and click on the Add New Datasource option. Here is the configuration that adds the data source for the InfluxDB database:

Modifying the Configure and ConfigureServices methods in Startup

Up to now, we have set up Ubuntu and the InfluxDB database on our machine. We also set up the InfluxDB data source and added a dashboard through the Grafana website. Next, we will configure our ASP.NET Core web application to push real-time information to the InfluxDB database.

Here is the modified ConfigureServices method that initializes the MetricsBuilder to define the attribute related to the application name, environment, and connection details:

public void ConfigureServices(IServiceCollection services)
{
  var metrics = new MetricsBuilder()
  .Configuration.Configure(
  options =>
  {
    options.WithGlobalTags((globalTags, info) =>
    {
      globalTags.Add("app", info.EntryAssemblyName);
      globalTags.Add("env", "stage");
    });
  })
  .Report.ToInfluxDb(
  options =>
  {
    options.InfluxDb.BaseUri = new Uri("http://127.0.0.1:8086");
    options.InfluxDb.Database = "appmetricsdb";
    options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
  })
  .Build();
  services.AddMetrics(metrics);
  services.AddMetricsReportScheduler();
  services.AddMetricsTrackingMiddleware();         
  services.AddMvc(options => options.AddMetricsResourceFilter());
}

In the preceding code, we have set the application name app as the assembly name, and the environment env as the stage. http://127.0.0.1:8086 is the URL of the InfluxDB server that listens for the telemetry being pushed by the application. appmetricsdb is the database that we created in the preceding section. Then, we added the AddMetrics middleware and specified the metrics containing the configuration. AddMetricsTrackingMiddleware is used to track the web telemetry information which is displayed on the dashboard, and AddMetricsReportScheduled is used to push the telemetry information to the database.

Here is the Configure method that contains UseMetricsAllMiddleware to use App Metrics. UseMetricsAllMiddleware adds all the middleware available in App Metrics:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
  if (env.IsDevelopment())
  {
    app.UseBrowserLink();
    app.UseDeveloperExceptionPage();
  }
  else
  {
    app.UseExceptionHandler("/Error");
  }
  app.UseStaticFiles();
  app.UseMetricsAllMiddleware();
  app.UseMvc();
}

Rather than calling UseAllMetricsMiddleware, we can also add individual middleware explicitly based on the requirements. Here is the list of middleware that can be added:

app.UseMetricsApdexTrackingMiddleware();
app.UseMetricsRequestTrackingMiddleware();
app.UseMetricsErrorTrackingMiddleware();
app.UseMetricsActiveRequestMiddleware();
app.UseMetricsPostAndPutSizeTrackingMiddleware();
app.UseMetricsOAuth2TrackingMiddleware();

Testing the ASP.NET Core App and reporting on the Grafana dashboard

To test the ASP.NET Core application and to see visual reporting on the Grafana dashboard, we will go through following steps:

1. Start the Grafana server by going to {installation_directory}\bin\grafana-server.exe.
2. Start bash from the command prompt and run the sudo influx command.
3. Start another bash from the command prompt and run the sudo influx command.
4. Run the ASP.NET Core application.
5. Access http://localhost:3000 and click on the App Metrics dashboard.
6. This will start gathering telemetry information and will display the performance metrics, as shown in the following screenshots:

The following graph shows the total throughput in Request Per Minute (RPM), error percentage, and active requests:

Here is the Apdex score colorizing the user satisfaction into three different colors, where red is frustrating, orange is tolerating, and green is satisfactory. The following graph shows the blue line being drawn on the green bar, which means that the application performance is satisfactory:

The following snapshot shows the throughput graph for all the requests being made, and each request has been colorized with the different colors: red, orange, and green. In this case, there are two HTTP GET requests for the about and contact us pages:

Here is the response time graph showing the response time of both requests:

If you liked this article and would like to learn more such techniques, go and pick up the full book, C# 7 and .NET Core 2.0 High Performance, authored by Ovais Mehboob Ahmed Khan.

Get to know ASP.NET Core Web API [Tutorial]

How to call an Azure function from an ASP.NET Core MVC application

ASP.NET Core High Performance