Take control of threading issues impacting the performance of your Java Web API application

In this post, learn how to monitor Java threads to understand the specific lines of code in your application that are causing performance issues.

Thread-related issues can adversely affect performance of a Web API application in ways, which are often hard to diagnose and tricky to resolve. Keeping a clear picture of a thread’s behavior is essential to achieving optimal performance. In this post, I’ll show you how to use Parasoft SOAtest‘s Load Test JVM Threads Monitor to view the threading activity of a JVM with graphs of vital statistics and configurable thread dumps that can point to the lines of code responsible for performance loss caused by inefficient thread use. Parasoft SOAtest’s load testing module enables you to convert any functional tests into load and performance tests.

We’ll follow a hypothetical Java development team that runs into a few common threading issues while creating a Web API application, and diagnose some common thread-related performance issues. After that we’ll look at more complex examples of the real applications. (Note that some suboptimal code in the examples below has been added intentionally for the purposes of demonstration.)

The Bank application

Our hypothetical Java development team embarked on a new project – a REST API banking application. The team set up a continuous integration (CI) infrastructure to support the new project, which includes a periodic CI job with Parasoft SOAtest‘s load testing module to continuously test performance of the new application. (For more details on how to set up automated performance testing, see my previous post, Load and Performance Testing in a DevOps Delivery Pipeline.)

Bank application version 1: Race conditions in the initial implementation

The Bank application code is starting to grow, and the tests are running. However, the team noticed that after implementing a new transfer operation, the Bank application started having sporadic failures under higher load. The failure comes from the account validation method that occasionally finds a negative balance in overdraft protected accounts. The failure in account validation causes an exception and an HTTP 500 response from the API. The developers suspect that this may be caused by a race condition in the IAccount.withdraw method when it is called by different threads processing concurrent transfer operation on the same account:

13: public boolean transfer(IAccount from, IAccount to, int amount) { 14:    if (from.withdraw(amount)) { 15:        to.deposit(amount); 16:        return true; 17:    } 18:    return false; 19: }

Bank application version 2: Adding synchronization

The developers decide to synchronize access to accounts inside the transfer operation to prevent the suspected race condition:

14: public boolean transfer(IAccount from, IAccount to, int amount) { 15:     synchronized (to) { 16:         synchronized (from) { 17:             if (from.withdraw(amount)) { 18:                 to.deposit(amount); 19:                 return true; 20:             } 21:         } 22:     } 23:     return false; 24: }

The team also adds the JVM Threads Monitor to the Load Test project that runs against the REST API application. The monitor will provide graphs of deadlocked, blocked, parked and total threads and will record dumps of threads in these states.

The code change is pushed to the repository and gets picked up by the CI performance testing process. The next day, the developers find that the performance test failed overnight. The Bank application stopped responding shortly after the start of the transfer operation performance test. Inspecting the JVM Threads Monitor graphs in the Load Test report quickly shows that there are deadlocked threads in the Bank application (see Fig 1.a). The deadlock details were saved by the JVM Threads Monitor as a part of the report and show the exact lines of code responsible for the deadlock (see Listing 1.b).

Fig 1.a – Number of deadlocked threads in the application under test (AUT).

DEADLOCKED thread: http-nio-8080-exec-20     com.parasoft.demo.bank.v2.ATM_2.transfer:15     com.parasoft.demo.bank.ATM.transfer:21     ...     Blocked by:         DEADLOCKED thread: http-nio-8080-exec-7             com.parasoft.demo.bank.v2.ATM_2.transfer:16             com.parasoft.demo.bank.ATM.transfer:21             com.parasoft.demo.bank.v2.RestController_2.transfer:29             sun.reflect.GeneratedMethodAccessor58.invoke:-1             sun.reflect.DelegatingMethodAccessorImpl.invoke:-1             java.lang.reflect.Method.invoke:-1             org.springframework.web.method.support.InvocableHandlerMethod.doInvoke:209

Listing 1.b – Deadlock details saved by JVM Threads monitor

Bank application version 3: Resolving deadlocks

The Bank application developers decide to resolve the deadlock by synchronizing on a single global object and modify the transfer method code as follows:

14: public boolean transfer(IAccount from, IAccount to, int amount) { 15:     synchronized (Account.class) { 17:         if (from.withdraw(amount)) { 18:             to.deposit(amount); 19:             return true; 20:         } 21:     } 22:     return false; 23: }

The change resolves the deadlock issue of Version 2 and the race condition of Version 1, but the average transfer operation response time increases more than five-fold from 30 to over 150 milliseconds (see Fig. 2.a). The JVM Threads Monitor BlockedRatio graph shows that from 60 to 75 percent of the application threads are in BLOCKED state during the load test execution (see Fig. 2.b). The details saved by the monitor indicate that application threads are blocked while trying to enter the globally synchronized section on line 15 (see Listing 2.c).

   BLOCKED thread: http-nio-8080-exec-4     com.parasoft.demo.bank.v3.ATM_3.transfer:15     com.parasoft.demo.bank.ATM.transfer:21     com.parasoft.demo.bank.v3.RestController_3.transfer:29     ...     Blocked by:         SLEEPING thread: http-nio-8080-exec-8             java.lang.Thread.sleep:-2             com.parasoft.demo.bank.Account.doWithdraw:64             com.parasoft.demo.bank.Account.withdraw:31

Listing 2.c – Blocked threads details saved by the JVM Threads Monitor

Bank application version 4: Improving synchronization performance

The development team searches for a fix that would resolve the race condition without introducing deadlocks and compromising the responsiveness of the application and after some research finds a promising solution with the use of java.util.concurrent.locks.ReentrantLock class:

19: private boolean doTransfer(Account from, Account to, int amount) {             20:    try { 21:        acquireLocks(from.getReentrantLock(), to.getReentrantLock()); 22:        if (from.withdraw(amount)) { 23:            to.deposit(amount); 24:            return true; 25:        } 26:        return false; 27:     } finally { 28:         releaseLocks(from.getReentrantLock(), to.getReentrantLock()); 29:     } 30: 

The graphs in Fig. 3a show response times of Bank Application transfer operation of version 4 (optimized locking) in red graph, version 3 (global object synchronization) in blue graph and version 1 (unsynchronized transfer operation) in green graph. The graphs indicate that the transfer operation performance has improved dramatically as a result of the locking optimization. The slight difference between the synchronized (red graph) and unsynchronized (green graph) transfer operation is an acceptable price for preventing the race conditions.

Fig 3.a – transfer operation response time of Bank application Version 4 (red), Version 3 (blue) and Version 1 (green).

Real World Examples

Example 1: Growing Application Response Time

The Bank Application examples above serve the purpose of demonstrating how to resolve typical isolated cases of performance degradation caused by threading issues. The real world cases may be more complicated – the graphs in Fig. 4 show an example of a production REST API application whose response time kept growing as the performance test progressed. The application response time grew at a lower rate for the first half of the test and at a higher rate in the second half (see Fig 4.a). In the first half of the test the response time growth correlated with total time application threads spent in the BLOCKED state (see Fig 4.b). In the second half of the test the response time growth correlated with the number of application threads in PARKED state. The stack traces captured by the Load Test JVM Threads Monitor provided the details: one pointed at a synchronized block, which was responsible for excessive time spent in BLOCKED state. The other pointed at the lines of code that used java.util.concurrent.locks classes for synchronization, which was responsible for keeping threads in the PARKED state. After these areas of code were optimized, both performance issues were resolved.

Example 2: Occasional Spikes in Application Response Time

The Load Test JVM Threads Monitor can be very helpful in capturing details of rare thread related problems, especially if your performance tests are automated and run regularly*. The graphs in Fig. 5 show a production REST API application that had intermittent spikes in average and maximum response times (see Fig. 5.a).

Such spikes in application response time can often be caused by suboptimal JVM garbage collector configuration, but in this instance a correlating spike in the BlockedTime monitor (see Fig. 5.b) points to threads synchronization as the source of the problem. The BlockedThreads monitor helps even more here by capturing the stack traces of the blocked and the blocking threads. It is important to understand the difference between the BlockedTime and the BlockedThreads monitors.

The BlockedTime monitor shows accumulated time that JVM threads spent in BLOCKED state between monitor invocations, while the BlockedThreads monitor takes periodic snapshots of JVM threads and searches for blocked threads in those snapshots. For this reason, the BlockedTime monitor is more reliable at detecting threads blockage but it just alerts you that thread blocking issues exist. The BlockedThreads monitor because it takes regular threads snapshots may miss some thread blocking events, but on the plus side, when it captures such events, it provides detailed information of what causes the blocking. For this reason, whether or not a BlockedThreads monitor will capture the code related details of a blocked state is a matter of statistics, but if your performance tests run on a regular basis, you will soon get a spike in the BlockedThreads graph (see Fig. 5.c), which means that blocked and blocking thread details have been captured. These details will point you to the lines of code responsible for the rare spikes in the application response time.

Creating performance regression controls

The Load Test JVM Threads monitor besides being an effective diagnostic tool, can also be used to create performance regression controls for thread related issues. After you have discovered and fixed such performance issue, create a performance regression test for it. The test will consist of an existing or a new performance test run and a new regression control. In case of Parasoft Load Test that would be a QoS Monitor Metric for a relevant JVM Threads Monitor channel. For example, for the issue described in example 1 (Fig. 4) create a Load Test QoS Monitor metric that checks for time that application threads spent in BLOCKED state and another metric that checks the number of threads in PARKED state. It is always a good idea to create named threads in your Java application, this will allow you to apply performance regression controls to a name-filtered set of threads.

Using Java Threads monitor in automated performance tests

The following table provides a summary of what Threads Monitor channels to use and when:

Conclusion

Parasoft’s JVM Threads Monitor is an effective diagnostic tool for detecting thread-related JVM performance issues as well as creating advanced performance regression controls. When combined with SOAtest‘s Load Test Continuum, the JVM Threads Monitor helps eliminate the step to reproduce performance issues by recording relevant thread details that point to the lines of code responsible for poor performance, and help you improve both application performance as well as developer and QA productivity.