Identifying locking issues with jstack

As SOA Suite is a heavily multi-threaded Java application, where locks are used to synchronize resources. If the threads encounter issues sharing these locks, then the impact on performance can be severe.

jstack is a HotSpot JVM tool that allows you to view thread dumps, which can be a useful way of identifying locking problems in your application.

Getting ready

You will need SOA Suite installed for this recipe, and will need permission to execute the domain control scripts, as well as the JVM tools. This recipe assumes that your SOA Suite application is running, and under a normal load.

The tools jps and jstack are included with the HotSpot JVM. If you're using JRockit, then see the There's more.. section of this recipe for the alternative command to use. For brevity, this recipe assumes that you have the relevant JVM bin directory on your command line path. If you do not, simply use the fully qualified paths to the relevant bin directory.

How to do it…

1. Use JPS to determine the process ID of your SOA Suite server; see step 1 of the Identifying New Size Problems with jstat recipe.

2. Use the jstat command to create a thread dump for the process. We specify the -l parameter to display additional information about locks, and redirect the output to a file, so we can inspect it more easily:

   jstack -l <pid> >jstack-output.txt

   This will generate a file called jstack-output.txt. Alternatively kill -3 <pid> can be used on Linux or Ctrl + Break in the console in Windows.

3. Open jstack-output.txt in your favorite text editor. There are two main types of locking problem we are looking for. The first is a deadlock, where two threads each own one or more locks, but are both waiting on each other's locks. This will look something similar to the following screenshot:

   

   Note that each thread is waiting on the lock held by the other thread. If you see thread deadlocks, this generally requires that you redesign your application to acquire locks in the correct order.

   The second type of problem we are looking for is lock contention. This occurs when many threads are all waiting to get access to the same lock. When this occurs, you will see many threads all waiting on the same lock. This usually also requires application code changes to resolve.

How it works…

The jstack tool comes with the HotSpot JVM, and is used to produce stack dumps for a running JVM. The output of stack dumps contains information on any locks held by each thread, which we can use to diagnose common locking-related problems.

There's more…

There are two ways of performing locking in a Java application. The first is by using the synchronized keyword, and is the most common. Locks obtained using this method appear in the thread dumps as threads marked as - locked <object reference>, and threads waiting to enter a synchronized block that they do not own the lock to are marked as – waiting on <object reference>. The second mechanism for obtaining locks is using the java.utilconcurrent.locks package, and locks obtained in this way do not appear in a standard thread dump. However, by specifying the -l parameter to jstack, we are able to view the information on which threads hold locks. This information is displayed in the locked ownable synchronizers section below each thread dump.

Free tools are available to examine lock dumps, but we will not discuss them in this book.

The techniques in this recipe can be replicated on the JRockit JVM with the command jrcmd <pid> print_threads, where pid can be identified using JRockit's jps in the same way as with the HotSpot JVM.