ASP.NET site performance: reducing long wait times

Measuring wait times

We can use a number of ways to find out which external requests are most frequent and how long the site has to wait for a response:

• • Run the code in the debugger with breakpoints around each external request. This will give you a quick hint of which external request is the likely culprit. However, you wouldn't do this in a production environment, as it only gives you information for a few requests.

• Use the Trace class (in the namespace System.Diagnostics) to trace how long each request takes. This will give you a lot of detailed information. However, the overhead incurred by processing all the trace messages may be too high to use in a production environment, and you would have to somehow aggregate the trace data to find which requests are the most frequent and take the longest.
• Build performance counters into your code that record the frequency of each request and the average wait time. These counters are light-weight, and hence, can be used in a production environment. Also, you can readily access them via perfmon, along with the counters provided by ASP.NET, SQL Server, and so on that you have already come across.

The remainder of this section focuses on performance counters. Also, performance counters are a convenient way to keep an eye on off-box requests on a day-to-day basis instead of as a one-off.

Windows offers you 28 types of performance counters to choose from. Some of these are esoteric, others extremely useful. For example, you can measure the rate per second that a request is made, and the average time in milliseconds that the site waits for a response. Adding your own custom counters is easy, and you can see their real-time values in perfmon, along with that of the built-in counters.

The runtime overhead of counters is minimal. You have already come across some of the hundreds of counters published by ASP.NET, SQL Server, and Windows itself. Even if you add a lot of counters, CPU overhead would be well under one percent.

This section describes only three commonly used counters: simple number, rate per second, and time. A list of all types of counters with examples of their use is available at http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecountertype.aspx?ppud=4.

To use the counters, you need to follow these three steps:

1. Create custom counters.
2. Update them in your code.
3. See their values in perfmon.

Creating custom counters

In this example, we'll put counters on a page that simply waits for one second to simulate waiting for an external resource.

Windows allows you to group counters into categories. We'll create a new category "Test Counters" for the new counters.

Counter NameCounter TypeDescriptionNbr Page HitsNumberOfItems6464 bit counter, counting the total number of hits on the page since the website started.Hits/secondRateOfCountsPerSecond32Hits per secondAverage WaitAverageTimer32Time taken by the resource. Inspite of the name, it is used here to simply measure an interval, not an average.Average Wait Base*AverageBaseUtility counter required by Average Wait.

*The text says there are three counters, but the table lists four. Why? The last counter, Average Wait Base, doesn't provide information on its own, but helps to compute the value of counter Average Wait. Later on, we'll see how this works.

There are two ways to create the "Test Counters" category and the counters themselves:

• Using Visual Studio: This is relatively quick, but if you want to apply the same counters to for example your development and production environments, you'll have to enter the counters separately in each environment
• Programmatically: Because this involves writing code, it takes a bit longer upfront, but makes it easier to apply the same counters to multiple environments and to place the counters under source control

Creating counters with Visual Studio

To create the counters in Visual Studio:

1. Make sure you have administrative privileges or are a member of the Performance Monitor Users group.
2. Open Visual Studio.
3. Click on the Server Explorer tab.
4. Expand Servers.
5. Expand your machine.
6. Right-click on Performance Counters and choose Create New Category.
7. Enter Test Counters in the Category Name field.
8. Click on the New button for each of the four counters to add, as listed in the table you saw earlier. Be sure to add the Average Wait Base counter right after Average Wait, to properly associate the two counters.
9. Click on OK when you're done.

This technique is easy. However, you'll need to remember to add the same counters to the production machine when you release new code with new custom counters. Writing a program to create the counters is more work initially, but gives you easier maintenance in the long run. Let's see how to do this.

Creating counters programmatically

From a maintenance point of view, it would be best to create the counters when the web application starts, in the Global.asax file. However, you would then have to make the account under which the application pool runs part of the Performance Monitor Users group.

An alternative is to create the counters in a separate console program. An administrator can then run the program to create the counters on the server. Here is the code.

using System;
using System.Diagnostics;
namespace CreateCounters
{
class Program
{
static void Main(string[] args)
{

To create a group of counters, you create each one in turn, and add them to a CounterCreationDataCollection object:

CounterCreationDataCollection ccdc = new
CounterCreationDataCollection();

Create the first counter, Nbr Page Hits. Give it a short help message and the counter type. Now, add it to the CounterCreationDataCollection object:

CounterCreationData ccd = new CounterCreationData
("Nbr Page Hits", "Total number of page hits",
PerformanceCounterType.NumberOfItems64);
ccdc.Add(ccd);

Add the second, third, and fourth counters along the same lines:

ccd = new CounterCreationData("Hits / second",
  "Total number of page hits / sec",
  PerformanceCounterType.RateOfCountsPerSecond32);
ccdc.Add(ccd);

ccd = new CounterCreationData("Average Wait",
"Average wait in seconds",
PerformanceCounterType.AverageTimer32);
ccdc.Add(ccd);

ccd = new CounterCreationData("Average Wait Base", "",
PerformanceCounterType.AverageBase);
ccdc.Add(ccd);

Now, it's time to take the CounterCreationDataCollection object and make it into a category. Because you'll get an exception when you try to create a category that already exists if there already is a category with the same name, delete it now. Because you can't add new counters to an existing category, there is no simple work-around for this:

if (PerformanceCounterCategory.Exists("Test Counters"))
{
PerformanceCounterCategory.Delete("Test Counters");
}

Finally, create the Test Counters category. Give it a short help message, and make it a single instance. You can also make a category multi-instance, which allows you to split the category into instances. Also, pass in the CounterCreationDataCollection object with all the counters. This creates the complete category with all your counters in one go, as shown in the following code:

PerformanceCounterCategory.Create("Test Counters",
"Counters for test site",PerformanceCounterCategoryType.
SingleInstance,ccdc);
}
}
}

Now that you know how to create the counters, let's see how to update them in your code

Updating counters in your code

To keep things simple, this example uses the counters in a page that simply waits for a second to simulate waiting for an external resource:

using System;
using System.Diagnostics;
public partial class _Default : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{

First, increment the nbrPageHits counter. To do this, create a PerformanceCounter object, attaching it to the nbrPageHits counter in the Test Counters category. Then, increment the PerformanceCounter object:

PerformanceCounter nbrPageHitsCounter =
new PerformanceCounter("Test Counters", "Nbr Page Hits", false);
nbrPageHitsCounter.Increment();

Now, do the same with the Hits/second counter. Because you set its type to RateOfCountsPerSecond32 when you generated it in the console program, the counter will automatically give you a rate per second when viewed in perfmon:

PerformanceCounter nbrPageHitsPerSecCounter =
new PerformanceCounter("Test Counters", "Hits / second", false);
nbrPageHitsPerSecCounter.Increment();

To measure how long the actual operation takes, create a Stopwatch object, and start it:

Stopwatch sw = new Stopwatch();
sw.Start();

Execute the simulated operation:

// Simulate actual operation
System.Threading.Thread.Sleep(1000);

Stop the stopwatch:

sw.Stop();

Update the Average Wait counter and the associated Average Wait Base counter to record the elapsed time in the stopwatch.

PerformanceCounter waitTimeCounter = new
PerformanceCounter("Test Counters", "Average Wait", false);
waitTimeCounter.IncrementBy(sw.ElapsedTicks);
PerformanceCounter waitTimeBaseCounter = new
PerformanceCounter("Test Counters", "Average Wait Base",
false);
waitTimeBaseCounter.Increment();
}
}

Now that we've seen how to create and use the most commonly used counters, it's time to retrieve their values.

Viewing custom counters in perfmon

Accessing your custom counters goes the following way:

1. On the server, run perfmon from the command prompt. To open the command prompt on Vista, click on Start | All Programs | Accessories | Command Prompt. This opens the monitor window.
2. Expand Monitoring Tools and click on Performance Monitor.
3. Click on the green "plus" sign.
4. In the Add Counters dialog, scroll down to your new Test Counters category.
5. Expand that category and add your new counters. Click on OK.

   

6. To see the counters in action, run a load test. If you use WCAT, you could use files runwcat_testcounters.bat and testcounters_scenario.ubr from the downloaded code bundle.