Using Stubs in Integration-Level Testing

Several months ago, one of our large customers, working on a safety-critical project being developed according to IEC 61508, contacted us to ask for help optimizing the developers’ productivity. The problem the customer was facing was due to the amount of noise in the test results generated by developers working concurrently on unit test cases, and configuring stubs for the needs of their specific test scenarios.

We learned that the way our customer was performing their unit testing is closer to integration testing. In their process, the tested units are not isolated from dependent components (other files in the project), and unit test cases are executed against the almost complete application, so all calls between functions in the projects are wired exactly the same way during unit testing as in the production builds.

This approach is not “classic” unit testing, as it borrows a lot from integration level testing. Still, it is very efficient for demonstrating good test coverage for requirements and source code.

For more detail on unit testing, visit parasoft.com/solutions/unit-testing

In this process, stubs are added only when a specific test scenario has to be simulated, typically a fault injection. Take, for example, the code below:

At the beginning of the function, there is an if statement that tests whether the buffer for samples was successfully allocated. Most of the test cases for this function were implemented without any stubs, as they are focused on the “regular” control flow, except the test case, which checks the behaviour of the function when buffer allocation fails. This test case requires a stub for the allocateSampleBuffer function to simulate the failure.

Once the stub is added, it will be consistently applied for the tested code. A user working on the “allocation failure” test case will have an easy way now to install a special call-back function into the stub, which will simulate desired effect (allocation failure, or do nothing since by default stub returns a null pointer which is expected for the test case). But all other test cases require attention right now because a stub configuration has to be added for them to avoid unwanted changes in the control flow.

Of course, developers can go back and reconfigure their test case to account for the stub, but it means extra time spent analyzing the reason of failure, preparing a dedicated callback function for the stub, and removing the noise in the testing process – which was the customer’s main concern when they contacted us.

So we added a special option for stubs in Parasoft C/C++test‘s 10.4.3 release that makes it much easier to generate automatic or user stubs. The new option is available in two places:

• For automatically generated stubs: Test Configuration -> Execution -> Symbols (tab)

• For user stubs (and auto stubs as well): Stubs Settings panel of the Stubs View

With the “insert call to original function” option checked, Parasoft C/C++test changes the default way the stubs are generated. The change is in a stub behavior when no call-back is installed. The stub generated with the new option will act as a proxy and call the original function definition unless the user provides a test case-specific call-back function that is meant to perform alternative activities. Stubs generated without the new option (including legacy stubs) will not try to call the original symbol in the default situation, and if there is no test case-specific call-back function installed, the stub will do nothing and just return a “default” value, such as a null pointer or zero numerical value.

To make sure the difference is clear, let me quickly compare the situation with the “insert call to original function” option both enabled and without it, for the case when the user did not provide a dedicated call-back:

• Here’s a stub generated for goo function, without the “insert call to original function” option. It works in the following way:

• Here’s how it looks for the same stub generated for goo function, but this time with the “insert call to original function” option checked:

As you can see, stubs added with the new option are transparent for the tested code and they simply perform the proxy call to the original definition, unless someone provides a call-back that implements desired alternative action.

An experienced engineer is likely to ask a question here like, “Ok, but what happens if there is no original definition available for the stubbed function? How does the stub behave in this scenario when I don’t provide a call-back that defines the alternative behavior and I don’t have an original definition available?”

Well, the beauty of this new functionality is that this kind of situation is automatically detected, and the stub will reconfigure itself at the test harness build time, not to call the original definition when no call-back is installed but rather return a safe default value.

This new functionality highly reduces the amount of interference between different team members when working concurrently on the test cases in this kind of “semi-integration” testing. A stub added by developer A will not change the behavior of the tested code for the test cases added by developer B. If developer B decides they need to configure an alternative action for the stubbed function for one of the test cases, they can simply create a test case-specific call-back function that implements desired alternative logic for the stubbed function, and install this call-back in the existing stub as a part of test case configuration.