Functional Regression Testing with SOAtest as part of a Continuous Integration Process

Achieve velocity while safeguarding your application against regressions

Continuous Integration (“CI”) is a well-understood and (at this point) well-adopted practice. It is a necessary first step to significantly improve the velocity of application delivery.

Continuous Integration allows developers to push their changes into a “master” branch of source code, with a single developer potentially pushing many changes to the master branch throughout a single day. To ensure that the master branch is pristine, buildable, and high-quality, it is critical to test after every change, since it serves as the golden copy of source code for the application.

(If you’re interested in more about continuous integration, I recommend Martin Fowler’s older but still interesting article here on the history of integration in software development and the benefits/best practices of CI.)

Today we’re going to focus on how to configure Parasoft SOAtest to execute functional regression testing as part of a Continuous Integration process. In this article, I’ll describe the steps to configure SOAtest with Jenkins, a popular automation platform. We’ll use the open-source Parabank application and deploy it using Docker to keep things simple.

How will this work?

The diagram below explains what we’ll be setting up in this article. It’s best read from left to right.

In short, Jenkins will check out a repo from Github which contains the SOAtest project called “Parabank” that contains REST tests. Jenkins will also pull a Docker image called parasoft/parabank from Docker Hub. This image contains not only Parabank, but also Tomcat and the correct Java runtime environment.

Jenkins will then run an instance of this Parabank image (called a “container”). Afterwards, Jenkins will tell SOAtest to run the tests we pulled from Github so that we can validate our Parabank instance.

Now, this isn’t quite in the spirit of true continuous integration (since I’m giving you a pre-built application), but I wanted to use Docker to save you the trouble of having to build Parabank with Maven and having to install and configure Tomcat/Java.

A more realistic/real-world diagram of CI is provided below. The Developer checks source code into Github. Now we want to test that the application is still in a good state even after the Developer’s changes.

The source code change in Github triggers a build in Jenkins, and Jenkins kicks off an automated Maven build (which executes JUnit tests). If all unit tests pass, then the packaged application (parabank.war) is deployed onto Tomcat. SOAtest then begins to execute functional “black box” tests.

Only after the unit tests pass (during the Maven build) and the functional “black box” tests pass (during SOAtest execution) are the developer’s original changes considered good.

Let’s get to the steps necessary to configure the process in the first diagram!

Configuring SOAtest and Jenkins

Prerequisites:

• A Windows 10 machine capable of running Docker (other operating systems will work, but some of the commands I provide in this article may have different syntax). This machine must have internet connectivity.
• Create a localsettings.properties file somewhere on your machine. Copy the contents of my sample file into it. If you have a node locked license, add it to line 5. If you use a license server, set line 6 to true and add your license server hostname on line 3. We will need the path to this file later.
• Jenkins 1.651 or newer installed on the same machine
• SOAtest 9.10 or newer installed and on the PATH (so that you can invoke soatestcli from any directory) of the same machine.
• Docker installed and on the PATH of the same machine.

Steps:

1. Log into Jenkins in your web browser (Jenkins is typically deployed at a URL like http://<JENKINS_HOST_IP>:8080/jenkins)
   
2. We’re going to start by installing some Jenkins plugins. Select “Manage Jenkins” on the left and then “Manage Plugins” in the new menu that appears.
   
   
3. Under the “Available” tab, select and install these plugins:
   a. “Parasoft Findings”
   b. “Git plugin” (version 3.30)Select “Install without Restart” and then, on the Installation page, check the box which says “Restart Jenkins when installation is complete and no jobs are running”
4. Return to the main Jenkins menu form step 1. On the left, select “New Item”
5. Provide the name “Parabank Deploy and Test” and select “Freestyle” project and then OK
6. In the configuration menu which appears, scroll down to Source Code Management and select Git. Add this URL to the Repo URL field: https://github.com/sdebrosse/soatest-automation-example.git . All other fields can be left with their default values.
7. Scroll to the bottom of the page and under Build add an “Execute Windows batch command” build step (if you’re using Linux, select “Execute shell” instead):
   
8. Copy the contents of the script here and paste it into the new build step field. You need to change the values of the two variables at the top of the script to reflect the path to your own localsettings.properties file and where you want a temporary workspace to be created (SOAtest will create this workspace during the testing process). Comments in the script explain what is happening on each line:
9. That’s it. We’re now ready to execute our Jenkins job! Make sure you first close all open instances of SOAtest. Then select Save at the bottom of the configuration menu, and click “Build Now” on the left:
   
10. You can click on the running job on the left and then view live console output:
    
    The log will say “SUCCESS” at the end if everything worked correctly. This means you’ve successfully pulled a SOAtest test project from Github, deployed a Docker container with Parabank and executed tests against this Parabank instance. At the end of the process, we automatically stopped the Parabank container and deleted our temp_workspace to clean up our environment.But wait a second, you may have noticed from looking at the logs that we had some test failures…Yes, a few of our tests against Parabank failed. If we want our Jenkins build to fail due to SOAtest test failures, we add the -fail flag when we invoke soatestcli. Like this:soatestcli.exe -fail -data %TEMP_WORKSPACE_PATH% -resource /Parabank -config “builtin://Demo Configuration” -localsettings %LOCALSETTINGS_PATH%

    If you open the test in the SOAtest Desktop UI, you’ll find out that this failure is primarily a test data/test environment configuration issue. Our loan processor declined a loan that it should have approved.

Where do we go from here?

Test environment configuration and test data are the biggest obstacles to reliable automated testing. In future articles, I’ll explore how a technology called service virtualization can help ensure that we always have the exact environment configuration we need in order to run our tests reliably at any time – this will segue us from Continuous Integration into the world of Continuous Testing.