Identifying performance problems with jstack
Jstack is a JVM tool that will generate thread dumps for a running JVM. We can use this as a way to do some lightweight profiling of our running SOA Suite application, to identify where performance bottlenecks are.
Getting ready
You will need the SOA Suite installed for this recipe, and will need permission to execute the domain control scripts, as well as the JVM tools. This recipe assumes that your SOA Suite application is running under a normal load.
The tools jps and jstack are included with the HotSpot JVM. If you're using JRockit then see the There's More… section of this recipe for the alternative command to use. For brevity, this recipe assumes that you have the relevant bin directory on your path. If you do not, simply use the fully qualified paths to the relevant bin directory.
How to do it…
Use JPS to determine the process ID of your SOA Suite server; see step 1 of the Identifying New Size Problems with jstat recipe.
Execute a slow performing use case on your SOA Suite server.
While the use case is executing, use the
jstatcommand to create a thread dump for the process. We redirect the output to a file to inspect it more easily:jstack<pid> >jstack-output-1.txt
This will generate a file called
jstack-output-1.txt. Alternatively,kill -3 <pid>can be used on Linux or Ctrl + Break in the console in Windows.Repeat step 3 a number of times, changing the number of the generated file each time, to create a sequence of thread dumps. I suggest aiming to create thread dumps at intervals of one or two seconds, for a period of 30 seconds, or the amount of time that it takes to execute your slow use case.
Open the output files in your favorite text editor.
To identify the poor performing code, we are looking for thread stacks that appear frequently across the files. These will either be operations that are executed multiple times, or operations that are executed once but take a long time to complete.
How it works…
A thread dump is a snapshot of what the every thread in the application is doing at a particular moment in time. By taking multiple thread dumps over a period of time, we can build up a picture of where the application is spending its time. This approach is essentially how a sampling profiler works, so we are effectively doing manual profiling of the application.
It is common to find that the cause of poor performance is waiting on data from external systems, such as a database or web service. In this case, what we will see in our thread dumps is that there will be many instances where a thread is in JVM code that is reading data from a socket. This is the thread waiting on a response from the remote system, and we know that we should target the external system in order to improve the performance.
We can quickly look through large numbers of stacks by noting that, in general, threads of interest to us will be those that are running the application code. We can identify the threads of interest, as they will generally have two properties; they will be executing the code from the packages that are used in your application, and they will have stack traces that are longer than those of threads that are currently idle.
There's more…
What we are doing here is essentially what a sampling profiler tool would do, although we are doing it manually. While there may be cases in which a sampling profiler would be a preferable way of doing this, we have found that doing the profiling manually gives a number of benefits. Firstly, it uses only tools that already exist on the server—there is nothing that needs installing, and given that we may be doing this on a live system, this usually means less paperwork. Secondly, we have more control over how and when we generate the thread dumps, which helps reduce the overhead on the application.
If we find this technique is something that we do frequently, it may be worth writing a script that will generate the thread dumps for us. We have found this to be a good compromise between using a sampling profiler and manually sampling, using thread dumps.
It is worth noting that WebLogic can report Java threads as "stuck" if they run for a longer period of time than a configurable timeout. This itself is not indicative of a problem, as long-running activities (such as database or file polling) can be valid things for application components to do. Observing thread dumps will tell us definitively whether we need to take further action.
Generating heap dumps is also a good technique for investigating performance issues, and free tools such as the Memory Analyzer Toolkit can generate reports on object allocation and aid with detecting memory leaks. We do not discuss heap dumps explicitly in this book, because SOA Suite is an Oracle supported application, so issues with the core components themselves should be raised with Oracle support, but there are plenty of good resources online to get use this technique if you desire.
See also
The Identifying performance problems using VisualVM on HotSpot, Identifying performance problems using JRMC on JRockit, and Using JRockit Flight Recorder to identify problems recipes
