Forrester: What’s the Role of Testing in Agile and DevOps?

From Diego Lo Guidice‘s Continuous Delivery Conference keynote:

“The term everyone is using is DevOps, but I think it should really be DevTestOps. Testing is really a crucial for achieving quality @ speed.

Forrester’s 2015 State of Agile Development study revealed the following about leading firms’ testing organization structure:

• 61% – Testers are allocated full-time to project teams
• 57% – Testers are members of an integrated delivery team
• 41% – Performance and load testers are centralized
• 36% – Have a Testing Practice Center of Excellence with specialized skills
• 35% – Performance and load testers are in project teams
• 29% – Have a Testing Center of Excellence where testing is managed and executed
• 28% – Automation architects are shared across projects
• 18% – Testing is managed and executed in the Testing Center of Excellence

The study also revealed that only 13% of respondents used one Agile technique and interestingly, 54% mixed different Agile techniques and 27% deliberately mixed Agile and non-Agile techniques. Clearly, there are hybrid approaches emerging in the real world. The most popular methods are Scrum, iterative, Kanban, DevOps and Lean. The most popular hybrid method is waterfall and Agile mixed, “WaterScrumFall.” Despite the popularity, there are still barriers to adoption, the study found the following to be the top four:

• Lack of strong leadership
• Behavioral change required in team members
• Lack of skilled product owners
• Lack of dedicated cross functional teams

Some positive news from the study indicated that testers are becoming peers to developers and first class citizens of the Agile process. As an organization’s test capability matures, testing becomes a critical function and testers need to be part of the team as peers. Clearly, the leading organizations are making testers part of an integrated team. They don’t have a test center of excellence decoupled from development. Testers are truly part of the team.

If you’re doing Agile, you need to be able to complete all the testing in your iteration, and your “definition of done” needs to cover testing. Ideally, within each iteration, you’re doing unit testing, functional testing, and even performance testing—this is all shifting to the left. You also need to ensure that you’re using the right test data. You can have a great testing process and great development process, but if you’re not using the right test data, quite often the quality will be poor.

It’s also interesting to look at the testing practices in use in leading organizations who are practicing agile:

• 82% – Manual testing
• 78% – Unit testing
• 67% – User acceptance testing in sprints
• 66% – Functional regression test automation
• 54% – Functional test automation
• 43% – Exploratory testing
• 42% – Test data management – Production data sub-settings, etc.
• 32% – Test data management – Masking, Obfuscation
• 25% – Test data management – Synthetic Generation
• 24% – Test-Driven Development (TDD)
• 19% – Service virtualization
• 14% – Behavior-Driven Development (BDD)

More Continuous Delivery Keynote Excerpts

Want to learn more from Diego Lo Guidice’s Continuous Delivery Conference keynote? Check out these earlier blogs and videos:

• Forrester: Why Continuous Testing is Vital for Quality @ Speed:  How testing needs to evolve in order to enable quality at speed for Agile and DevOps.

• Forrester: Barriers and Best Practices for Scaling Agile: The top 3 reasons Agile fails, how “Agile Expert” firms overcome common barriers, and Agile best practices.

• Forrester: What’s the Role of Testing in Agile and DevOps?: The importance of integrating testers and testing practices into your development teams.

• Test Automation, API Testing, and Service Virtualization in DevOps: Why test automation, API testing, and service virtualization are critical to agile development initiatives as organizations shift to continuous delivery, Agile, and DevOps.

Forrester’s research points to the following benefits from Agile and state of the art testing methods. Agile isn’t about making software easy or cheaper to developer but rather achieve better outcomes with the amount of time, effort and money poured into the development:

• Alignment between the business and IT
• Delivery of solutions that meet requirements
• Ability to make corrections during development
• Increase agility by removing the testing bottleneck