Gartner: Why DevOps Requires Continuous Testing

New Gartner research on DevOps and Continuous Testing guides Agile teams to achieve the optimal balance of speed and quality in this era of “Continuous Everything.”

In today’s economy, businesses create a competitive edge through software — and every company is essentially a software company. Now that rapid delivery of differentiable software has become a business imperative, software development teams are scrambling to keep up. In response to increased demand, they are seeking new ways to accelerate their release cycles — driving the adoption of agile or lean development practices such as DevOps.

Yet, based on the number of software failures now making headlines on a daily basis, it’s evident that speeding up the SDLC opens the door to severe repercussions.

Agile Development Teams Must Reinvent the Software Testing Process

Organizations are remiss to assume that yesterday’s practices can meet today’s process demands. There needs to be a cultural shift from testing an application to understanding the risks associated with a release candidate. Such a shift requires moving beyond the traditional “bottom-up” approach to testing, which focuses on adding incremental tests for new functionality. While this will always be required, it’s equally important to adopt a top-down approach to mitigating business risks. This means that organizations must defend the user experience with the most likely use cases in the context of non-functional requirements — continuously.

In order for more advanced automation to occur, we need to move beyond the test pass/fail percentage into a much more granular understanding of the impact of failure: a nuance that gets lost in the traditional regression test suite. Continuous Testing is key for bridging this gap. Continuous Testing brings real-time assessments, objective go/no-go quality gates, and continuous measurements to refine the development process so that business expectations are continuously met.

Ultimately, Continuous Testing resets the question from “are you done testing?” to “is the level of risk understood and accepted?”

A Business-Focused Approach to Continuous Testing

How does this business-focused approach to Continuous Testing work? At a high level, you situate a broad set of automated defect prevention and detection practices that serve as “sensors” throughout the SDLC — continuously measuring both the product and the process. If the product falls short of expectations, you don’t just remove the problems from the faulty product. You also consider each problem found an opportunity to re-examine and optimize the process itself—including the effectiveness of your sensors. This establishes a defect-prevention feedback loop that enables you to incrementally improve the process.

In terms of DevOps, the benefits of Continuous Testing include:

• Business stakeholders always have real-time access to feedback on whether their expectations are being met, enabling them to make informed decisions.
• At the time of the critical “go/no go” decision, there is an objective assessment of whether the organization’s specific expectations are satisfied—reducing the business risk of a fully-automated Continuous Delivery process.
• Defects are eliminated at the point when they are easiest, fastest, and least costly to fix—a prime principle of being “lean.”
• Continuous measurement vs. key metrics means continuous feedback, which can be shared and used to refine the process.

This notion of “continuous quality” is central to achieving the expected ROI from DevOps, agile, and other lean initiatives.

New Gartner Research: DevOps and Continuous Testing

DevOps Requires Continuous Testing by Gartner Research provides software development leaders recommendations for transforming the SDLC to achieve the optimal balance of quality and speed in this new era of “Continuous Everything.” Read it to learn:

• Why Agile teams must reinvent the software testing process to meet today’s demands
• Why it’s time to evolve from asking “Are you done testing” to determining “Is the level of risk understood and accepted?”
• How Continuous Testing reduces the risk of a fully-automated Continuous Delivery process—while enabling continuous improvement and reducing waste in the SDLC
• Recommendations for applying “continuous quality”, code review, and static analysis tools to increase the ROI of DevOps initiatives