What is Continuous Testing and How to Bring It Into Your Organization

Businesses are focusing more and more on customer experience as a part of their go-to-market strategy, and a key piece of the customer’s experience is their ability to traverse through your software in a quick and seamless way.

To reduce the risks of a bad customer experience, organizations are doubling down on their quality initiatives, and the software development industry is embracing continuous testing as a mainstream activity.

What is continuous testing?

Continuous testing is a principle of software testing, in which all of your tests are executing all of the time, providing continuous feedback into the quality and health of your applications. But in order to achieve continuous testing, organizations must first adopt test automation. There are many types of test automation, spanning the breadth of an application starting at the UI layer, through the middleware systems, and even back-end systems. Understanding how to bring in these different types of test automation practices as efficiently as possible enables you to move towards the path of continuous testing.

Building the Right Test Automation Strategy

The now widely-accepted test pyramid (popularized by Michael Cohen and Martin Fowler) identifies the optimal strategy for the different types of testing activities. At the base, representing the largest quantity of tests, we want to establish broad coverage of unit and API tests, which are easiest to run in automation, but often require technical skill to build. Getting to the top of the testing pyramid, you find automated UI tests and manual tests. We want this type of testing activity at the top because it’s the only way to ensure the customer experience.

Most people focus at the bottom and the top of the testing pyramid with an “automate what’s manual” approach for UI testing and a “developers should test” mentality for unit testing. While these practices are important, focusing on the top and bottom creates a gap in the middle, at the API layer, between the UI and the code. But this gap is only going to continue to become more and more problematic, with a recent survey from Programmable Web predicting over 22,000 publicly available APIs by 2021. API testing is more critical than ever, needs to become an integrated part of the continuous testing strategy that you’re building.

3 Steps to Enable Continuous Testing with Test Automation

I previously discussed how to choose the best API testing solution for your organization’s unique needs, calling out key features that you might need depending on your industry and application. It’s helpful to start with an API testing solution that can grow with you as your API testing maturity grows. Once you’ve chosen a tool, how do you get started? Here are the three critical steps to achieve functional test automation faster and enable your continuous testing.

Step 1: Establish broad test coverage of your existing APIs for automation

To create a broad set of automated tests, you can build scriptless tests from web recordings and API contracts, and then use those tests to continuously validate the health of your APIs, ensuring that the APIs are working as they were designed. Think of this as unit testing for APIs but without the gruntwork – not only is this a valuable testing technique, but it’s also one of the earliest types of functional test validation you can do because service contracts for APIs are usually one of the first things that are written, as new features or functionalities are created.

As an example, let’s say that the team has added some new functionality in our banking application. As a first step, the development team has published a new service definition. In this case, a swagger document was generated. The new service that was being added was the RequestLoan service. This service takes a series of inputs and responds with a loan provider for the new loan. To test this service, I could consume the Swagger YAML, and create a series of clients for each individual operation.

One of those clients would be the request loan service. I could create a series inputs, both positive and negative, to validate that the service behaved appropriately. I could then repurpose these tests for regression purposes.

Of course, while this type of testing is extremely valuable, it’s only half of the API testing puzzle because it doesn’t validate how the APIs are actually being used. Enter step 2.

Step 2: Bridge the gap between UI and API

The second part of the API testing strategy is being able to model human usage of your application into complete API test scenarios. You can start to bridge that gap in the testing pyramid by taking advantage of artificial intelligence to augment your ability to understand what’s actually happening behind the scenes when users are navigating your application, and interpret those behind-the-scenes transactions as API calls. This type of testing allows you align the user experience with your critical API tests.

AI is a key component of the strategy because we can reliably use AI to help us break down these communications into relationships and patterns, to understand the business rules of how our applications are being tested. We can bring this together with our unit-level API testing to get broad coverage of our API spectrum.

Continuing my example from earlier, you will notice that the request loan service requires inputs from other areas of my application. Specifically:

Now, while I could provide customerID and from accountAccountID arbitrarily, I really need them to exist in my application. So I will need to create a dynamic scenario where I first query the individual user to obtain the customer ID and account ID so that I can hand that information off to the request loan service and ensure that a dynamic scenario works as described. Ensuring that I’m working with real dynamic data ensures that that behavior that exists as a result of APIs interacting with each other can be fleshed out.

These types of techniques will allow us to shift-left the API testing practice and create broad coverage of our applications in the earliest stages possible. Once achieved, there is a third critical component of this practice, which is all about our ability to understand and adapt to change.

Step 3: Ensure confidence through a maintainable change management process

I’ve spoken to so many people about their functional testing initiative that fell flat once their application changed. This is a common occurrence because testers spend the majority of their time building rich and great API tests, only to have them break when the application’s APIs change. This can have the cumulative effect of reducing confidence in the API testing strategy because testers are spending a large portion of their time maintaining their API tests instead of building new value.

Change management is a key piece of any functional testing strategy, and AI can be a critical enabler here as well. By automatically scanning service definitions (yes, the same service definitions used to initially create the test cases) to identify when your APIs have changed, you can understand when you’ll be impacted, and then build a template for migrating existing services to the new version.

Going back to the first part of our banking application example, I made the statement that a new service was added to my application. This actually represents API change. Since I created the first round of baseline tests using the service definition, I can now compare the different versions of the service definition to each other, to not only identify what has changed but to build a map to update my existing test cases:

Upon reviewing the change template, it’s easy to see that not only has a new service been added, but many of my existing services have been re-factored as well. In the image above you will notice that get customers has a series of new fields that have been added. Using a change management workflow, you can proactively identify service changes while at the same time managing the updating of existing test cases, so that you can recover from the change as quickly as possible.

This is arguably the most important practice one must establish when building their functional testing strategy, and having an understanding and a commitment to quality from the beginning will help you and your organization adopt this practice

What about flaky test environments?

So it would be easy to stop there, with your excellent test automation enabling continuous testing. But say you’ve spent a good amount of time building this rich and powerful functional testing strategy, you run your tests in your environment as a part of your automated nightly continuous testing process, and upon reviewing the results you see that a large portion of your tests have failed due to a system that’s outside of your control. Does this mean that your tests were bad? Are you now being held responsible for systems that are technically out of scope of your testing?

This is not an uncommon story. We know that functional tests can only be as effective as the test environments in which they execute. Unstable, unavailable, or just plain flaky test environments can reduce the return on investment we get from our functional testing tools. So I must, at least briefly, mention one of the best ways to stabilize your test environments, and that is: service virtualization.

Not to be confused with virtual machines (that’s hardware virtualization), service virtualization enables you to actually simulate the services that are communicating between different pieces of hardware. For example, think of an application calling a database. Do you actually need that database in your test environment? What if it doesn’t have the data that you need? With service virtualization, you can record the transactions with the database and then use that recording to create a simulated version of that database, with all of the behavior you want for your test environment. But of course, it doesn’t stop at just databases – it can be any type of service, such as a SOAP or REST API, even TCP and Microservices.

As a part of building a sustainable API testing strategy, you’ll need to build a sustainable service virtualization strategy, and that starts with answering questions like:

• What services are good candidates for virtualization?
• How are virtual services created?
• How can I maintain a virtual environment?
• How can I deploy a virtual environment as a part of my continuous testing strategy?

Service virtualization is a key enabler for a sustainable continuous testing strategy, but understanding where and when to bring it in and how to be as effective as possible is key to success.

Now what?

So now that you have a better understanding of how to integrate API testing as a part of your continuous testing strategy, the next step is to get going! Get engaged with an API testing tool vendor, and start with step one. Understanding the end goal as you begin will help you make smart choices along the way. Happy continuous testing!