Why DevOps Needs Continuous Testing

Continuous testing isn’t just about automation. True continuous testing for DevOps aligns test activities with business value. The goal of continuous testing in support of DevOps is to eliminate meaningless tests, and produce value-added tasks that truly drive the development organization towards a successful release.

In response to today’s demand for “Continuous Everything,” the software delivery conveyer belt keeps moving faster and faster. But considering that testing has been the primary constraint of the software delivery process, it’s unreasonable to expect that simply speeding up a troubled process will yield better results.

A great analogy for I Love Lucy fans is the infamous scene of Lucy and Ethel at the candy factory, struggling to keep pace as the conveyer belt starts putting out chocolates faster and faster.

Now that rapid delivery of differentiable software has become a business imperative, software development teams are scrambling to keep up. In response to increased demand, they are seeking new ways to accelerate their release cycles—driving the adoption of agile or lean development practices such as DevOps.

But based on the number of software failures now making headlines on a daily basis, it’s evident that speeding up the SDLC opens the door to severe repercussions.

Organizations are remiss to assume that yesterday’s practices can meet today’s process demands. There needs to be a cultural shift from testing an application to understanding the risks associated with a release candidate. Such a shift requires moving beyond the traditional “bottom-up” approach to testing, which focuses on adding incremental tests for new functionality. While this will always be required, it’s equally important to adopt a top-down approach to mitigating business risks. This means that organizations must defend the user experience with the most likely use cases in the context of non-functional requirements—continuously.

In order for more advanced automation to occur, we need to move beyond the test pass/fail percentage into a much more granular understanding of the impact of failure: a nuance that gets lost in the traditional regression test suite.

Continuous Testing, which involves automatically executing a set of tests designed to verify whether business expectations for functionality, security, reliability, and performance are satisfied, is key for bridging this gap. It shifts the paradigm from a pure bottom-up approach to a hybrid model in which top-down testing is leveraged to protect the expected user experience from changes introduced as the application evolves. Ultimately, Continuous Testing resets the question from “are you done testing?” to “is the level of risk understood and accepted?”

Testing: The Elephant in the Room

As organizations begin to accelerate the SDLC, process bottlenecks will become evident. One of the bottlenecks that continues to plague SDLC acceleration is testing. At best, testing has been considered inevitable overhead — a “time-boxed” event that occurs sometime between code complete and the target release date. At worst, organizations have pushed quality processes to post-release, forcing a “real-time customer acceptance test.”

Testing has always been the elephant in the room. Psychologically, the malleable nature of software has given organizations an excuse to defer investment in testing. However, this deferral results in technical debt. Over time, technical debt compounds, escalating the risk and complexity associated with each release.

Another obstacle to SDLC acceleration is the lack of a coordinated, end-to-end quality process. If trustworthy and holistic quality data were collected throughout the SDLC, then more automated decision points could optimize downstream processes.

Continuous Testing: How It Helps

Continuous Testing provides a real-time, objective assessment of the business risks associated with an application under development. Applied uniformly, it allows both business and technical managers to make better trade-off decisions between release scope, time, and quality.

Continuous Testing isn’t just more automation, it’s a larger reassessment of software quality practices — driven by an organization’s cost of quality, balanced for speed and agility. Ultimately, Continuous Testing can provide a quantitative assessment of risk and produce actionable tasks that will help mitigate these risks before progressing to the next stage of the SDLC.

Figure 1: Mitigate risks before they progress to the next stage of the SDLC

For example, assume that a retailer knows that very specific user experience factors related to the application make the consumer more likely to add items to their e-commerce shopping cart. These metrics, which are defined in business language and measured automatically, are the exact same benchmarks used to accept a software release candidate. The question that the test suite should answer is how application modifications impact this set of business metrics. The goal is to bring the business expectations and business language to the forefront of the validation process—ultimately, protecting the user and the brand.

When it comes to software quality, we are confronting the need for true process re-engineering. Continuous Testing is not a “plug and play” solution. As with all process-driven initiatives, it requires the evolution of people, process, and technology. We must accommodate the creative nature of software development as a discipline, yet we must face the overwhelming fact that software permeates every aspect of the business—and software failure presents the single greatest risk to the organization.

Summary

Given the business expectations at each stage of the SDLC, Continuous Testing delivers a quantitative assessment of risk as well as actionable tasks that help mitigate these risks before they progress to the next stage of the SDLC. The goal of continuous testing in support of DevOps is to eliminate meaningless tests, and produce value-added tasks that truly drive the development organization towards a successful release.