Vulnerability Management: Navigating the Deep Dark Pit of Version Numbers

One of the first things any security practitioner will tell you to do is keep your software up to date. It’s the number one way to protect against exploits targeting known vulnerabilities. In fact, most attacks these days don’t use new or novel attack methods, or even recently discovered vulnerabilities to succeed. They often use vulnerabilities that are years old!

Now while it’s simple to say that everyone should just run the most recent versions of operating systems and packages, actually implementing this quickly becomes painful for a number of reasons.

Risk Drives the Decision to Patch

Let’s begin by talking about risk. The idea is that if you update to the most recent version of a software package, you eliminate risk. However, you may not really be at risk, even if you have a vulnerable package version installed. This is either because your implementation of that package makes it impossible to exploit, or you have other mitigations in place that otherwise prevent the exploit from working.

The other possibility is accepted risk. This is where you have a vulnerable package installed but choose not to upgrade it, usually due to interoperability reasons: if you upgrade to the latest version, it may break integrations with other dependent software and cause you considerable grief. In this case, you accept that the cost of being vulnerable is lower than the cost of changing your other dependent software, and you’re accepting the possible consequences. This is often an unfortunate reality.

In both cases, this process and the decision to install or pass on an update should be well documented.

Blog Flowchart.png

How Do We Know What’s Vulnerable?

Mitre CVE offers a database of Common Vulnerability and Exposure IDs (CVE-ID.) According to the website, a CVE Identifier (CVE ID) number reservation allows vulnerability researchers and vendors to include CVE IDs in the initial public announcement of a newly discovered vulnerability, and ensures that the CVE ID is instantly available to all CVE users and makes it easier to track vulnerabilities over time. This is synchronized with the National Vulnerability Database (NVD).

National Vulnerability Database

The National Vulnerability Database (NVD) attaches a CVSS score. (CVSS stands for Common Vulnerability Scoring System.) The CVSS score uses a mathematical model to score vulnerabilities by risk. The problem here is that determining the level of risk for a vulnerability in your environment largely depends on the particulars of your specific environment. For example, if a vulnerable package exists in your environment but is only accessible by your engineering team, the risk is far lower than if it is accessible by the entire internet. The point here being that context is everything. Of course you should still patch it, but where that lives on your list of priorities might change considerably.

The other problem with CVSS is the ambiguity and how people assign scores to the underlying fields. While there is a model used to derive these scores, many of the variables are subjective and depend on the point of view of the analyst who is populating them. This is why it’s often recommended to review vulnerabilities based on the package name, focusing on critical or network software, in addition to reviewing based on CVSS.

Even so, CVEs are the best way to identify known vulnerable or at risk software in your environment, leveraging CVSS and knowledge of your environment to determine where you should focus your time on patching.

Versions, Versions, Versions

This brings us to the most difficult part of understanding whether your packages and OS are up to date. Version numbers are a mess. This is in large part because the package developer’s version number will usually differ from the OS’s version number of the same package. On top of that, that operating system version number will likely differ from other distributions. So while I might check the actual developer’s website to make sure I’m up to date, the version number provided probably won’t be the same as the version number of that very same software that’s installed on my instance.


If you’re not familiar, a vulnerability was discovered in 2014 that affects the OpenSSL cryptographic software library. Specifically, it allows remote attackers on the internet to eavesdrop on communications that should otherwise have been impossible, and to potentially steal data and impersonate users and services. CVE-2014-0160 allows attackers to get access to encryption keys.

Obviously everyone wanted to patch as soon as possible to avoid being compromised, but to do this you needed to verify whether you’re running the most recent version of OpenSSL.

The vulnerable versions, according to OpenSSL, are 1.0.1 through 1.0.1f. However, if you’re running Red Hat Enterprise Linux version 6, those versions are 1.0.1 through 1.0.1e-16.el6_5.4. But if you’re running Red Hat Enterprise Linux version 7, they’re actually 1.0.1 through 1.0.1e-34.el7. Now if you’re a Debian user, that changes to 1.0.1e-2 through 1.0.1e-2+deb7u5, but you’ll still need to upgrade five additional packages to close this vulnerability on your system. If you’re running Ubuntu, well, good luck, because from what we could tell, the documentation is still unclear over two years later. Now to add insult to injury, if you simply run the openssl version command, you’ll get an even different set of results!

If you really don’t want to sleep at night, apply this same problem to potentially every software package you have installed.

Threat Stack Vulnerability Assessment

As a security, development, and operations company, closing vulnerabilities is near and dear to our hearts. In an effort to deal with this problem, we set a few of our developers loose on it. Because of their hard work, Threat Stack offers a unique feature that not only sorts out the version problem for you, but correctly identifies versions across multiple operating systems, versions, and packages. Because Threat Stack is installed on the host operating system, we have local visibility into the patch level of the entire system, as opposed to traditional scanners that try to get visibility from the outside in. This accurately answers the two most difficult questions related to this problem: which version do you actually have installed, and is the version vulnerable?

Screen Shot 2016-11-04 at 1.51.58 PM.png

To accomplish this, Threat Stack sources from multiple areas such as the NVD, email lists, website scraping, and more, and then normalizes this information before checking to see if your instances contain vulnerabilities. We link you to the appropriate resources so you can conduct the proper research to discover how these bugs affect you. Threat Stack even allows you to suppress certain vulnerabilities if, after conducting your research, you decide that you’re not vulnerable due to your implementation, or you’re willing to accept the risk of not patching. You can even create notes to provide to auditors explaining why you’re not going to patch.

Screen Shot 2016-11-04 at 2.02.13 PM.png

In Conclusion . . .

Software bugs are unavoidable, but patching shouldn’t be so hard. Here at Threat Stack we want to provide threat and vulnerability management and continuous visibility into your infrastructure so you can focus on making your organization successful, rather than hunting for version numbers. Our vulnerability assessment feature is just one way that Threat Stack can help.