Unit Test Coverage Metrics

Independent of the coverage granularity, there are a number of different coverage criteria that take different aspects of coverage into account.

The scope of the various coverage criteria will be illustrated using the Java method shown in Listing 2. The sample method does not perform any practical operation, but it is fairly simple and quite helpful for illustrating all coverage criteria discussed in this article. For reasons of simplicity, the tested method only has a single input parameter. However, since a method with two parameters can also be viewed as a method with one parameter that happens to be a pair or tuple, the concepts discussed in this article apply just as well to methods with multiple parameters.

Statement Coverage

The most basic coverage criterion is statement coverage (also sometimes referred to as “block coverage”). Complete statement coverage is achieved when each statement of a tested method is executed by at least one test case. For the sample method from Listing 2, two test cases, using the input values 0 and 1, are sufficient to achieve complete statement coverage. A corresponding JUnit test is shown in Listing 3. The body of the first if statement will be executed for both inputs.

Statement coverage does not take into account what would happen if the condition of the first if statement evaluated to false, in which case the body of the first if statement would not get executed. In practice, it is just as important to consider what happens when a certain piece of code is not executed. Testing of such situations needs to go beyond merely checking that every statement was executed.

public class Listing2    {      public static int[] method(byte input)      {          boolean condition1 = (input & 1) == 0;          boolean condition2 = (input & 2) == 0;          int output = -1;          if (condition2)          {              output++;          }          if (condition1)          {              return null;          }          else          {              return new int[output];          }      }  }

Listing 2: Sample Java method to demonstrate different coverage criteria

public class Listing3 extends junit.framework.TestCase  {      public void testMethod0()      {          int[] result = Listing2.method((byte)0);          assertNull(result);      }        public void testMethod1()      {          int[] result = Listing2.method((byte)1);          assertEquals(0, result.length);      }  }

Listing 3: Sample JUnit test case that provides 100% statement coverage for Listing 2

Branch Coverage

The coverage criterion that takes these situations into account is called branch coverage (also known as condition coverage or decision coverage). Complete branch coverage requires that all possible outcomes of a conditional expression are executed by a test case. For the sample method, this means that there needs to be a test input where condition2 (in the first if statement) evaluates to false and where the body of the first if statement is therefore skipped. To achieve 100% branch coverage for the sample method, you could add a third test case that uses an input value of 2. The test case that uses 0 as an input would be redundant in this scenario. If the test inputs are chosen carefully, you may be able to achieve complete branch coverage with the same number of inputs. However, this is not a general rule, and you might need more test cases to achieve branch coverage than you need to achieve statement coverage.

Path Coverage

Will 100% branch coverage guarantee that a piece of code is bug-free? Though it presents a significant improvement over mere statement coverage, complete branch coverage still provides no guarantee that a piece of code always behaves correctly.

The sample method in Listing 2 can actually throw a NegativeArraySizeException under certain circumstances. This problem occurs for all inputs where the body of the first if statement is skipped because condition2 evaluates to false and at the same time condition1 evaluates to false as well. In these cases, the code will attempt to allocate an array of size -1. Branch coverage does cover both alternatives of the second if statement, but it does not guarantee that the statement’s else branch is covered in combination with the body of the first if statement being skipped.

To provide coverage for such situations, we need to look at combinations of branches, also known as code paths. This coverage criterion is called path coverage. Path coverage requires not only that every possible branch is executed at least once, but also that every possible combination of branches is executed at least once.

Figure 1: UML activity diagram with possible code paths through the sample method

Figure 1 shows the control flow of the sample method from Listing 2 as UML activity diagrams. In each diagram, a different code path is highlighted in bold red. The sample method has a total of four different code paths. To achieve 100% path coverage, you need a minimum of four different test cases (for example, using the input values 0, 1, 2, and 3).