Streamline Embedded Software Testing With a CI/CD Workflow

Implementing a continuous integration and continuous delivery (CI/CD) workflow for embedded software development is becoming increasingly popular. Projects for embedded software are often constrained in ways application development is not. Besides the physical and computational constraints of the target hardware platform, there are constraints in the marketplace.

The embedded market tends to have unique requirements for safety, security, and privacy along with extremely long life cycles. The products can stay on the market for decades.

The Challenges of Embedded Software Testing

Automating testing for embedded software is challenging due to the necessity and complexity of initiating and observing tests on embedded targets. Also, software teams typically have limited access to the target hardware.

Software test automation is essential to make embedded testing workable on a continuous basis from the host development system to a target system. Testing embedded software is particularly time-consuming. Automating the regression test suite provides significant time and cost savings.

Essentials for Testing Embedded Software

Test results and code coverage data collection from the target system are essential for validation and standards compliance, as needed. And data collection is critical in test execution. It can be accomplished by recording and maintaining traceability between test cases, test results, source code, and requirements.

Integrated testing solutions, like Parasoft C/C++test, offer a test harness optimized to take minimal additional overhead for the binary footprint and provide it in the form of source code, where it can be customized if platform-specific modifications are required.

Automate Embedded Software Testing

There are solutions that provide dedicated integrations with embedded IDEs and debuggers to make the process of executing test cases smooth and automated. Parasoft’s testing solution for c/c++ software development supports the creation of regression testing baselines as an organized collection of tests and automatically verifies all outcomes. These tests run automatically on a regular basis to verify whether code modifications change or break the functionality captured in the regression tests.

If any changes are introduced, these test cases will fail to alert the team to the problem. During subsequent tests, Parasoft C++test will report tasks if it detects changes to the behavior captured in the initial test.

The parity of capabilities of remote target execution with host-based testing means that embedded software teams can reap the same benefits of automation as any other type of application development.

Automation is important but it needs to do more than keep the status quo. To improve software security, safety, and quality, more testing is needed within the CI/CD pipeline without further slowing progress.

Improve Test Automation to Optimize CI/CD for Embedded Software

The largest struggle teams face is how to improve testing efficiency while also meeting needs for safety, security, and quality without impacting schedules and costs. Inevitably, even with simple test automation (test execution and results), there are compromises needed to keep testing time reasonable.

The simple solution is selecting, with guesswork, what parts of the software to test as full system test suites are too time-consuming and expensive. Embedded software teams need to increase their testing while, at the same time, focus testing on exactly what is needed. Optimizing tests using smarter test automation takes the guesswork out of test creation and execution.

Optimize Testing With Code Coverage

In general, code coverage is a measurement of how much of the production code is executed while your automated tests are running. By running a suite of tests and looking at code coverage data, there’s a general sense of how much of the application is being tested.

There are multiple kinds of code coverage. For embedded systems, you need to be familiar with statement, branch, and MC/DC. For the strictest requirements, such as in safety-critical software, object code verification or assembly language code coverage may be required.

Collect & Analyze Code Coverage Metrics

Collecting and analyzing code coverage metrics is an important aspect of safety-critical embedded software development. Code coverage measures the completion of test cases and executed tests. It provides evidence that validation is complete, at least as specified by the software design.

It also demonstrates the absence of unintended behavior—code that isn’t covered by any test is a liability since its behavior and functionality are unknown. The amount and extent of code coverage depends on the safety integrity level. As expected, the higher the integrity level, the higher the rigor used and, inevitably, the number and complexity of test cases.

Here are examples of types of code coverage recommended.

• Statement coverage requires that each program statement be executed at least once (branch and MC/DC coverage encompasses statement coverage).
• Branch coverage ensures that each possible decision branch (if-then-else constructs) is executed.
• Modified condition/decision coverage (MC/DC) requires the most complete code coverage to ensure test cases execute each decision branch and all of the possible combinations of inputs that affect the outcome of decision logic. For complex logic, the number of test cases can explode so the modified condition restrictions are used to limit test cases to those that result in stand-alone logical expressions changing. Check out this tutorial from NASA.

Advanced unit test automation tools, like Parasoft C/++test, provide all of these code coverage metrics and more. C/C++test automates this data collection on host and target testing and accumulates test coverage history over time. This code coverage history can span unit, integration, and system testing to ensure coverage is complete and traceable at all levels of testing.

Increase Code Coverage With Automated Unit Test Case Creation

The creation of productive unit tests has always been a challenge. Functional safety standards compliance demands high-quality software, which drives a need for test suites that affect and produce high code coverage statistics.

Teams require unit test cases that help them achieve their coverage goals which are important even outside the realm of safety-critical software. Any code not covered by at least one test is shipping untested!

Increasing code coverage can be challenging. Analyzing branches in the code and trying to find reasons why certain code sections aren’t covered, continues to steal cycles from development teams.

Resolve Coverage Gaps

Teams can resolve coverage gaps in test suites using a coverage advisor. Parasoft discovered how to use advanced static code analysis (data and control flow analysis) to find values for the input parameters required to execute specific lines of uncovered code.

This analysis computes preconditions for function parameters, global variables, and external function calls required to execute a specific line of code. The coverage advisor presents a collection of solutions for the user selected lines of code. Presented values are used for creating new unit test cases. This functionality boosts the productivity of developers working on unit test cases to improve code coverage.

Each coverage solution includes:

• Required dependencies. Dependencies that need to be customized to cover the selected line. These may include function parameters, external function calls, global variables, local variables, and class members.
• Pre-conditions. Conditions that must be satisfied by the required dependencies to cover the selected line. Clicking a pre-condition navigates to the related code line.
• Expected coverage. Code lines that will be covered if all of the preconditions are satisfied.

Optimize Testing With Smart Test Execution

To accelerate testing in a continuous pipeline, smart test execution on a per-build basis is needed to reduce the set of tests required to be executed to address the risk that each new iteration has introduced. The analytics provided by test impact analysis are key to making testing focused on only what absolutely needs to be tested rather than the shotgun approach used otherwise.

Only through smart can data-based decision-making enable continuous testing. Focusing the development team on the minimum set of tests to ensure proper coverage at each iteration is the key to bring the agility back to Agile development methods.

Smart test execution in Parasoft C/C++test is extended with plugins for CI systems (Jenkins, TeamCity, Bamboo, and so on) for advanced functionality to help software development organizations reduce bottlenecks associated with running continuous builds. The same capabilities are available inside IDE environments with dedicated plugins that access a centralized coverage image through a REST API, and determine which tests need to be executed locally inside the IDE to verify all changed code.

On the development side, experienced developers might apply the proper structure in organizing their tests and run only a subset of them manually, but they still might not know which tests need execution to verify ALL changes.

Teams using CI might rely on nightly builds to execute all tests automatically overnight and get feedback the next day, but only if it’s possible to execute the total number of tests in under 12 hours.

Unfortunately, most software development teams are running their day-to-day operations accepting those unscalable testing practices. The situation becomes even more difficult when manual testing is involved. Traditional software development organizations are still following testing practices represented by an inverted pyramid, which emphasizes running manual tests over automated tests for one reason or another.

Smart test execution uses test impact analysis to trace the execution of manual tests against applications and associated, captured code coverage information with those tests. Similar technology is used for automated testing. This analysis figures out which manual tests need execution to access changed functionality delivered with every new build. Thus, smart test execution is critical at the developer and tester levels in their local IDEs. It enables them to focus the testing where it’s needed, removing guesswork and extra “just in case” work.

Summary

Continuous integration and continuous delivery are common place in embedded development. Migrating a waterfall process to CI/CD and Agile development pays off in risk reduction and quality and security improvements. However, testing remains a hurdle for CI/CD workflow adoption for embedded systems due to unique challenges in automating testing for them. Removing these obstacles with smart test execution and optimizing testing while improving code coverage, is the key to successful CI/CD adoption.