Towards Full Regression Test Coverage

As surprising as it may sound, even complete path coverage does not mean that your code always behaves correctly.

If you are using a test-driven development approach (TDD), you will be familiar with the idea of using unit tests as a kind of specification for the code under test. You first write a test that asserts what a new method is supposed to do, watch it fail, and add just enough implementation code to make it pass. If you show the test case from Listing 4 to a developer and ask for an implementation, the result might be a method that always returns 0, as shown in Listing 5. Interestingly, the developer would not even violate the spirit of TDD by doing so. There would be 100% path coverage and no test failure, but obviously the implementation would not work properly. In real life, these kinds of issues rarely appear in the form of simple methods that add two values, but usually take a more complicated shape. The fundamental problem is that a set of unit tests might provide full path coverage for a method, but at the same time might not provide a complete specification of that method. One typical scenario is that a developer optimizes the implementation of an existing method in such a way that all test cases still pass, but previously unasserted functionality is inadvertently removed or changed. This can happen even with 100% path coverage.

The illustrative code sample from Listing 2 would require 256 different test cases (for all possible values of the type byte) to achieve full regression testing coverage. Without the additional test cases, a developer could replace the original implementation with simplified code that returns correct results only for the test inputs 0, 1, 2, and 3, but simply throws an Unsupported¬Operation¬Exception for all other inputs. All existing test cases pass, yet a part of the original functionality is lost. Even a set of unit tests that provides full path coverage is not guaranteed to detect such a regression.

If path coverage is already difficult to achieve, full regression coverage can only be called a practical impossibility. The add method from Listing 4 takes two integer parameters, each of which can assume 232 different values. Full regression coverage for this method would require 264 (i.e., 232 × 232) test cases. With a clever parameterized testing approach, it would be possible to represent all these test cases in a compact fashion, but it would be impossible to execute all tests within a reasonable timeframe. Full regression coverage would detect even subtle problems like the infamous Pentium FDIV bug; unfortunately, it is completely impractical for all intents and purposes.

Failure to guarantee the proper functioning of something as simple as an adding method is not exactly trust-inspiring. So, short of creating test cases that exercise every possible combination of inputs, what can you do in practice to achieve a reasonable level of protection against regression failures? Clearly, the more different inputs you keep firing at a tested method, the better your regression coverage will be. The question is which inputs to choose for your test cases. Extreme values (or so-called “corner cases”) are a good choice. Test inputs for an int parameter could, for example, use Integer.MAX_VALUE, Integer.MAX_VALUE-1, 1, 0, -1, Integer.MIN_VALUE+1, and Integer.MIN_VALUE as test inputs. Similarly, a null reference, an empty string, and various strings containing special characters make good test inputs for a String parameter.

Test cases that aim at improved regression coverage often have a common test structure, but use different input values. In these situations, it may be helpful to use tools that support test case parameterization, which extracts the common test structure and separates the test data into an Excel spreadsheet or some other external data store. Another option for improving the regression coverage of a test suite is so-called perturbation testing, which applies minor modifications (such as adding or subtracting a small number) to input values from existing tests with the goal of exposing new interesting code paths and asserting previously unasserted behavior. Corner case values and perturbation testing are heuristic rather than systematic, but both can still increase regression coverage and can be applied manually or by automated tools such as Parasoft Jtest.

Table 1 provides a summary of the test cases that are necessary for achieving 100% statement, branch, path, and full regression coverage for the tested method from Listing 2.

Table 1: Test inputs to achieve 100% coverage of Listing 2