The Simple Way to Increase Java Unit Testing ROI

Learn how to reduce unit testing efforts by at least 50%. With these kinds of savings, software teams can increase their productivity without sacrificing quality and decrease delivery times considerably.

Capers Jones’ body of work is a great asset to software developers in any industry. He has spent his career studying the successes and failure of software projects, much of which culminated in 2011’s The Economics of Software Quality. Although enterprise Java developers may feel this collected information doesn’t appeal to them, I think they’re missing out on some valuable insight.

Here are some sobering statistics that apply as much to developers using Scrum and the latest software tools as they do to any other type of software development. As a whole, for every dollar the software industry spends on development, 50 cents of that dollar is spent on maintenance and finding and fixing bugs; however, most forms of testing only remove about 35% of bugs, leaving a majority of bugs in the software despite a team’s best efforts.

Jones was very much an early proponent of “shift-left testing,” although he may never have used the term. Finding and fixing defects (especially defects in requirements, analysis, and design) as early as possible in the software lifecycle is key to higher software quality. Interestingly, Jones stood out by making a good case against the cost-per-defect metric used by many vendors to calculate tool ROI – if anything, cost-per-defect undersells the ROI of automation tools and the shift-left testing effort.

Current Testing Success Rates

We already know that software developers are spending half of their development budget on finding and fixing bugs, and that current test methods are still leaving two thirds of the bugs in the software. Here’s some more interesting statistics:

• Even using the latest software development methods, defect removal rate is roughly 85% in the best projects. This means 15% of the defects are yet to be found and make their way into the final product.
• About 6% of test cases have bugs in the test cases themselves.
• In large projects, as many as 20% of regression tests are duplicates, which add to testing costs but do nothing to increase product quality.
• About 7% of bug fixes include new bugs. So, even as bugs are being resolved, there are new ones being introduced.

How Test Automation and the Parasoft Jtest Unit Test Assistant Can Help

As we’ve stated many times in the past, unit testing is necessary but a tedious requirement of software development. Test automation helps by removing much of the tedious processes from the developer, but test creation and maintenance remains one of the key problems facing Java developers when tackling unit testing of their code.  In a previous post, I outlined how to use Parasoft Jtest for automatic unit test creation, and how to increase unit test efficiency and outcomes while also reducing mocking complexity and test case maintenance. Continuing with this topic, let’s consider the economic benefits realized by automatic unit test creation and the impact it has on testing efforts.

In a recent survey conducted by Parasoft, we learned that a majority of developers are spending about 40% of their time on unit testing. Considering a two-week development sprint cycle consisting of ten days, four days are devoted to testing. It’s easy to see why testing can become a drag that slows down iterative and agile software development. In addition, the current testing success rate means that this amount of time still isn’t enough or more importantly, a way to decrease this time while improving outcomes is needed.

We’ve also been busy retrieving data from customers using Parasoft Jtest for Java testing and it’s very encouraging. Java development teams are seeing a decrease in their unit testing effort by a minimum of 50%. In other words, they can complete their four days unit testing in two days using Jtest and the Unit Test Assistant. This type of savings on a per-sprint basis is impressive but becomes more so when this is compounded over many sprints in a typical project. For example, if a typical project releases every three months, with six sprints of development, Jtest saves the equivalent of 1.2 sprints or 12 days of development effort. With these kinds of savings, software teams can increase their productivity without sacrificing quality and decrease delivery times considerably. Better quality and delivered on time (or even early)? These are serious economic benefits.

The Real ROI of Improved Quality

There’s more to return on investment for increased quality than the cost to fix a defect. Fixing a bug early in the lifecycle is cheaper, and doing so saves you money. Although this is one metric and even alone it’s sufficient to justify the investment in better quality, it actually undersells the ROI.

One of the leading causes of project delays is missed defects and security vulnerabilities that make their way into late stages of a product development cycle. Of course, it’s cheaper to find and fix these earlier because the development team still has the code fresh in their minds and have not moved on to the next iteration (or project for that matter).

Using only the cost-per-defect metric and approach to calculating ROI, consider the example above, with a team of 20 people working on a project with a loaded labor rate of $100 per hour. This team, using new test automation tools with all the benefits thereof (shift-left the identification of defects) and discovers 20 more defects than they did in the previous sprints. Finding and fixing these bugs early might require three hours per defect for a total $6,000. Finding and fixing these bugs later in the integration or system testing might triple the effort, for a cost of $18,000. Simplistically, for this sprint, the ROI is $12,000. Sounds great, right? However, this doesn’t factor in the 2 days of development time savings for the sprint for an additional $32,000 in savings and increased productivity.

Looking at the big picture, we can see that decreasing the development time for the entire release is the real money saver here, not the cost-per-defect. The real payoff for shifting left is achieving or beating project schedules and goals. Consider again the above example, but this time look at the ROI in terms of the whole development team finishing the release early by 12 days. For this team, that’s 12 days of 20 people which amounts to $192,000! Although this simple example seems obvious, it does point out that ROI from tools is realized at the team level, when products are delivered to market faster without sacrificing quality.

Conclusion

Traditional unit testing methods consume a large amount of software development time, and the outcomes from these approaches need improvement. Parasoft Jtest can help reduce the unit testing effort by 50%, which pays off handsomely in terms of quality and decreased sprint schedules.

The ROI from these tools is significant when you consider how much unit testing impacts the team and project as a whole. Unlike simple cost-per-defect analysis, finishing projects on time and meeting requirements on goals is the big payoff, and saving time and money while doing it makes it that much better.