Why Appsec Vulnerabilities Are Dismissed as “Theoretical” or “False”

This content was originally published on The Code Curmudgeon blog.

By Arthur Hicken, Chief Evangelist at Parasoft

In a previous post on theoretical Appsec vulnerabilities, I covered how “it’s theoretical” is misused by those who are trying to avoid fixing a security vulnerability or taking responsibility for it—for example, the Lenovo Superfish breach, Heartbleed, and airline wifi attacks.

The idea that a vulnerability is merely theoretical is not only ignorant but dangerous. Software exploits occur because bad actors operate by finding unexpected loopholes in a software system. Think of it this way – if you left your door unlocked is it a security issue? Or perhaps “If an unlocked door is never entered, is it really unlocked” if you’re a philosopher. One could contend that the risk is theoretical, but most of us would say that such a statement is ridiculous. (Props to those who live in an area where door security isn’t required.)

Software vulnerabilities are exactly like unlocked doors. They are not theoretical, they are actual whole or openings in your software than can and probably will be exploited. If we went to have secure applications, we must grow up and stop pretending that the vulnerabilities we’re facing aren’t real. We have to move a proactive preventative engineering based approach that treats vulnerabilities as real risks, only then can we be secure.

Nevertheless, people continue to justify ignoring the results from their security scanners. How do they rationalize this? In no particular order…

Noise

Noise, noise, noise! However you call it, whether it’s false positives, false alarms, theoretical problems, too many messages or anything else, noise is a big pain. Jeff says accuracy matters and indeed it does. But in my experience developers are awfully quick to pull the “noise” trigger. It would be fun to do a poll/quiz with a couple of real code samples and see what developers think. The basic application security quiz at the AppSec Notes blog but doesn’t expose the knowledge issues around recognizing real security vulnerabilities in actual code. In other words, would you recognize why a piece of code is dangerous if you saw it? I’d like to see the quiz that has both good code and bad code examples and the question “security problem or ok”. This is trickier than it sounds at first because you can’t just have bad code and fixed code, you need bad code and then suspicious code that would likely raise a false alarm. Now there’s a real test. If anyone knows of such a beast or wants to build one, make sure to let me know.

Noise can be made even worse if you have false objectives and metrics. For example if you base success on “number of issues found” you’re likely to find the greatest number of unimportant issues. There are better definitions of success discussed in the presentation AppSec Broken Window Theory that is worth reading. One quick example is to track success by the number of “flaws not found” in new applications.

One other point on noise, particularly in the area of what people like to call false positives. I have the “Curmudgeon Law of Static Analysis Tool Usefulness”. The more clever a static analysis tool is, the more likely its output will be perceived as a false positive. This is because it’s easy to accept a result you understand, but if a tool tries to tell you something you don’t know (IE the most useful result) you are least likely to accept it. That’s why I keep pushing on a) better presentation of results to explain WHY it matters and b) better training.

Bad Workflow or Process

This is closely related to the noise problem. If you have a bad workflow, process, or configuration you will find your security scanning painful. Sometimes this pain is noise, sometimes it’s extra effort, slow work or other symptoms. For example I’d seen people scan static analysis on legacy code where the corporate policy was “no changes unless there is a field-reported bug”. Scanning code you don’t intend to fix or looking for items you don’t intend to fix certainly contributes to noise, effort required and costs.

The takeaway is make sure your tools are configured optimally and the way they integrate into your process is not causing headaches. I could go on a long time on this topic alone but we’ll save that for another day. How can you tell if it’s causing headaches? Ask those who are using the tools and those who are required to address the output of the tools.

Another great read on this topic is Does Progress Come From Security Products or Process where Gunnar Peterson says “Process engineering is deeply unsexy. It is about as far removed from the glamour, fashion show world of security conferences as you can possibly imagine, but it’s where the actual changes occur.”

Prioritization

Again this is a category closely related to noise. Noise is a symptom of a lack of prioritization in your appsec findings. Ideally you’re storing all the output from your all the security tools in your arsenal in an intelligent data-driven system that will help you determine which items are most important. No one wants to spend weeks fixing something that is unlikely to happen and even if it did the consequences are minimal.

AppSec prioritization must take risk into account. Will this happen? Does this happen today? How hard is it to exploit? How hard is it to prevent? Plus all the other usual risk management questions. You know what to do, if you have too much noise from your tools, you need to put some intelligence behind it to get to what matters.

Note that when I say prioritization I don’t necessarily mean triage. Triage implies a human driven process (see bottleneck) that makes painful tradeoffs in a short time span rather than an orderly thought-out process. In the Broken Windows presentation I mentioned earlier Erik Peterson says “Triage != Application Security Program” and I wholeheartedly agree.

This is why I’m a fan of the attempt to develop comprehensive risk application frameworks. Some of these are the Common Weakness Scoring System (CWSS) that is managed at Mitre. CWSS proposes a way to properly weigh security findings in the context of your application, which should enable better automated prioritization. This goes hand-in-hand with the Common Weakness Risk Analysis Framework (CWRAF).

I know they’re far from perfect and in fact are frequently way too complicated and painful to use. Research in this area will ensure continued improvement and I expect it to ultimately become the critical driver in the appsec scanning space.

Test Security In

We’ve all heard the adage that you “can’t testing quality into an application”. This is based on Deming’s 3rd principle “cease dependence on inspection to achieve quality.” Early one many disagreed with Deming but today it’s an accepted truth that inspection alone will not solve your problems. Yet somehow in AppSec many believe that we can “test security into an application.” This attitude is easy to recognize because it results in security processes that are completely orthogonal to development, and are highlighted by all security activities being post-development QA-like activities. That method didn’t work for quality and won’t work for security.

In order to get ahead of the software security issues we have to start building secure software in the first place. There’s a great site sponsored by the US government called Build security in and it has lots of great information to get you going. Also take a look at the Building Security In Maturity Model. I realize that it’s easier to say “build it in” than it is to actually do and that building it in brings it’s own challenges. But like manufacturing and quality, it’s the only way to get long-term sustainable improvement.

Training

The SANS institute did a survey on application security programs and practices and found

“Although suppliers continue to improve the speed and accuracy of SAST tools and make them easier to use, developers need security training or expert help to understand what the tools are telling them, which vulnerabilities are important, why they need to be fixed and how to fix them.”

The report notes that only 26% of organizations had secure coding training that was working well or mandated. Further it reads

“A lack of knowledge and skills is holding back Appsec programs today, and it is preventing organizations from making real progress in Appsec in the future. The number one obstacle to success reported in this year’s survey is a shortage of skilled people, part of a bigger problem facing the IT security industry in general, as recent studies by Forrester Research13 and (ISC)214 show.

Training and education are needed to address this skills shortage—not just training more Infosec and Appsec specialists, but training developers and managers, too. Fewer than one-quarter of respondents have training programs that are ongoing and working well, and secure coding training ranks low in the list of practices that organizations depend on in their Appsec programs today. This needs to change. ”

OWASP talks about the importance of training

“From information security perspective, the holistic approach toward application security should include for example security training for software developers as well as security officers and managers…”

A survey in 2010 showed that security training was not being done and I suspect little has changed today.

“Nearly 80% of personnel at government agencies and contractors said their organization does not provide sufficient training and guidance for software security application development and delivery, according to a new survey.”

Summary

Tools alone and tool accuracy won’t fully answer the problems of AppSec. Again, from the Sans survey

“There aren’t any next generation tools or other silver bullets on the horizon that will solve the problem of secure software. Writing secure software is about fundamentals: thoughtful design, careful coding, disciplined testing and informed and responsible management. The sooner that organizations understand this—and start doing it—the sooner they will solve their security problems. ”

False alarms and accuracy are a huge problem. So is the lack of training. I wouldn’t want to argue that one is more important than the other, because I could easily find myself arguing both sides. Both are important. Until we have a perfect security tool, meaning no false alarms and equally important, no false negatives, training will play a necessary integral role in application security (appsec) and software security (swsec).