Modern Analytics for Modern Software Testing

Modern software testing practices deserve modern analytics, too. Learn how to go beyond your ordinary reports-and-dashboards tooling for software testing.

As my colleague, Chris likes to say… It’s 2018! You can expect more. Your testing tools should work for you. And he is absolutely right. We’ve been focused on empowering our software testing tools with modern technologies such as artificial intelligence and machine learning, so our customers can take all of that power and make moves.

To do so, you need some high-powered analytics. And that means, going beyond simple reports and dashboards. It means applying these modern technologies to reports and data. That’s right, the sexiest part of software testing!

Data granularity

So what do you need to revolutionize your analytics? It starts with the granularity of the data you are capturing, and the breadth of data you’re capturing. You can combine data from all of your testing techniques (from static analysis to manual testing to automated functional testing, etc.) and all tools that you’re using (not just the tools in the Parasoft ecosystem, but other tools you’re using, too). Then, you get flexible and customizable, so you can essentially measure anything your heart desires, for a comprehensive understanding of the health of your codebase, at a moment in time, or over time, across builds. And most importantly, because of all of this, you can quickly understand where to focus your limited resources to increase quality.

Increasing the quality of your code is, after all, the whole point. Right?

Beyond simple reports and dashboards

So what does it mean to go “beyond” simple reports and dashboards? I would argue that there are 5 main ways Parasoft helps you go beyond, into something so much more powerful than what we see in the analytics from other software testing tools.

The sheer amount of data you can capture

Parasoft captures an enormous amount of data across all of your testing practices – it’s truly unprecedented. Not just static analysis and code metrics, but also unit testing, functional testing, manual testing, code coverage, and code reviews. It aggregates all of this data together in order to give you a comprehensive view of the results of your testing practices.

In addition to aggregating all of this data, the data itself is collected in a granular way, enabling advanced computations and insights; flexibility and endless customization possibilities; and the ability to store a large amount of information over time. For example, because of the detailed data that you can keep on test execution, you can easily see, at any time, which tests executed against a specific file to achieve code coverage.

How you can navigate this data

After collecting your testing data, Parasoft provides an interactive framework for navigating and analyzing these results. Each report is made up of little customizable widgets, and when you click into one of them, you arrive at what is called an “Explorer” — an interactive UI for exploring your test data and understanding how changes in your codebase are changing code quality. Some of the explorers include the following:

• The Change Explorer lets you compare two builds and compare what files have been changed/edited/removed, and what static analysis violations map to those changes.
• The Code Coverage Explorer lets you navigate coverage from all of your testing practices, with traceability to the original tests that executed against the code.
• The Violation Explorer lets you view error reports such as MISRA rule violations in the context of the codebase, along with historical information, documentation, and specific details of the violation, so you can identify and trigger an issue remediation workflow.

Simplifying SDLC Workflows

As you explore deeper into the data within the Explorers, they also provide the framework to help you take action. The Explorers are connected via integrations to your existing software development tooling, so you can define actions for the team. For instance, you can create a task in JIRA, and augment that workflow by adding custom metadata to define things like priority, assignment, and additional context.

By connecting your analytics dashboard to your workflow management systems, you’re able to take action at the same time as you learn what actions need to be taken.

But these explorers and resulting workflows are only as powerful as the data and analytics behind them. And here is where PIE comes in, which you may have heard of, either as a popular fruit-filled dessert or as the award-winning Process Intelligence Engine (PIE for short), which calculates the advanced analytics and “derived metrics” that power Parasoft with true intelligence to inform your testing.

Performing the actual analytics

Parasoft’s Process Intelligence Engine (PIE) provides actionable insights based on highly flexible and machine-learning-enabled analytics. We’ve preconfigured existing calculations called PIE “slices” (see what we did there..), to empower your testing practices and subsequent decision-making, including:

• Test Stability Reporting – helps you quickly identify and prioritize real regressions in the code vs. chasing instabilities.
• Change-Based Testing – allows you to only re-test the tests that touched code changes.
• Modified Coverage – helps you focus on the coverage of only the code that’s changed, as opposed to the macro coverage that includes legacy code.
• Risky Code Changes – keys you in on which changes in the code are riskier than others (and you can customize your definition of “risk”).

These preconfigured PIE slices give you exceptional power and insight, but the part that makes this so neat is that you can create your own slices of the pie with whatever analytics you can dream up. The framework is fully customizable, so you can extend it in any way you want.

Compliance and policy enforcement

Finally, for establishing and enforcing policies, you can use Parasoft’s devoted Policy Center, which helps you stay organized with practice thresholds, aggregated policies, and quality gates that can be queried as part of your CI/CD pipeline.

On the compliance side of things, because of the aggregated data collection I mentioned above, you get a central location for managing and auditing your software testing in the context of whichever coding standards or rules you need to comply with. We have a few pretty awesome elements here:

1. Compliance packs (for MISRA, CWE, etc.) that include preconfigured widgets for your dashboards and preconfigured reports to make compliance and audits a lot easier.
2. The ability to take data from static analysis and display the results in context of your standard, without re-executing the analysis.
3. The ability to, when a specific build is ready for certification, archive all the data from the test runs, locking the build once certification is complete.

So what’s next?

There is a treasure trove of value to be gained from Parasoft’s modern analytics. Your software testing practice deserves it, and your team has everything to gain. And now, you can even integrate your software testing practices into your value stream. So go on and get your hands dirty. The analytic world is your dashboarding oyster.

“MISRA”, “MISRA C” and the triangle logo are registered trademarks of The MISRA Consortium Limited. ©The MISRA Consortium Limited, 2021. All rights reserved.