Automotive Software Engineering Defects on the Rise

“It takes dozens of microprocessors running 100 million lines of code to get a premium car out of the driveway, and this software is only going to get more complex.” — Robert N. Charette, IEEE Spectrum: This Car Runs on Code

The above quote was taken from an article written seven years ago—before Tesla’s Model S rolled off the assembly line, before Google started testing self-driving cars, and before integrated infotainment systems became standard on even lower-end models. Automotive software engineering for today’s cars is considerably more complex. Many models having more than 100 electronic control units (ECU), which are responsible for everything from monitoring a sensor to controlling an actuator. With so much complexity built into automotive software, it’s no wonder software-related defects are on the rise.

What is an ECU?

An ECU typically consists of a black-box assembly with an embedded microprocessor, memory, firmware, network connection, and circuitry for carrying out its function. Modern automobiles consist of an increasing number of ECUs connected by a sophisticated network communications bus that is more and more commonly using Ethernet.

The ECU’s embedded software might be designed and coded by the automobile manufacturer or supplied by a third-party in the vast automotive supply chain. When the ECUs are assembled into a finished vehicle, the entire system must be tested to ensure all components—not just core and safety-critical components—function correctly. In fact, the engine, brakes, airbags, and other features and functions won’t even work without software. That’s why, as Mr. Charette wrote in 2009, the car runs on code as much as it runs on fuel.

Ensuring Automotive Software Quality

The automobile manufacturer has complete legal and ethical responsibility for ensuring that the vehicle is safe to own and operate, which includes performing QA on many levels. For instance:

• Making sure that the brakes actuate when you press the pedal (functional safety)
• Preventing memory leaks that degrade performance of the engine oxygen sensors over time (non-functional safety)
• Preventing a hacker from exploiting the tire pressure monitoring system or accessing telemetry uploaded via cellular links (cybersecurity)

When a defect is found, the manufacturer can respond in several ways. If the defect is very minor and doesn’t pose a threat to the vehicle’s performance or safety, the manufacturer might simply choose to ignore it. If the defect represents a potential cost, but isn’t safety-critical, the firmware may be upgraded the next time the car is serviced. Alternatively, a newer version of a replaceable subassembly might include updated firmware. Some automakers have shifted to over-the-air (OTA) upgrades for compatible models and ECUs. In many cases, the manufacturer may not be required to disclose the software fix to the customer or to a regulatory agency.

The True Cost of Quality Includes the Cost of Recalls

If a severe defect with safety-critical implications is detected, the manufacturer may have to issue a recall. Recalls take place either voluntarily or at the “urging” of a regulatory agency. The recall would include software, which may have to be tested and approved before it is pushed out.

It is difficult to know for certain the cost of defective software in cars, but because recalls are publicly disclosed, we have some insights into costs and trends. From 2005 to 2012, there were 32 automotive recalls that included software fixes affecting 3.6 million vehicles. In the next 3.5 years–from 2012 through June 2015–there were 63 recalls associated with a software component affecting 6.4 million cars. In half the time, the impact of software defects nearly doubled.

Additionally, only about 5% of automotive recalls had a software-related component in 2011. In 2015, that number rose to almost 15%.

Those software-related recalls affect every part of the vehicle. From 2006 through 2015, there were software-related recalls for tire systems (e.g., tire pressure monitoring systems), the vehicle’s structure, latches and locks, fuel system, power train, vehicle speed control (i.e., regular or adaptive cruise control), visibility and exterior lighting, parking brakes, hybrid propulsion, engine and engine cooling, steering, airbags and electrical. For more about software-related recalls, see our previous post, “Quantifying the Risk of Automotive Software Failures: The SRR Warranty and Recall Report.”