What’s on the Box!? An In-depth Look At OS Package Management

Recently, I had the opportunity to help build out our vulnerability detection feature here at Threat Stack. I stepped into this project as I had many others; trying to understand the problem, thinking about the scale, how to break up the problem, etc. This problem is something developers rarely think about: the operating system. Sure, we have all done our fair share of apt and yum, but have you ever really taken a look into what gets installed on your computer? Have you ever noticed that when you do a dpkg -l, what you see is actually some strange take on semantic versioning that doesn’t seem to line up with what you see when you look at the version of that program using its version command? Me either, and let me tell you, it was not what I was expecting.

Let’s talk about Ubuntu. Ubuntu is a popular distribution of Linux. Based on Debian, it comes with some cool tools for installing and updating packages on your OS called dpkg and apt. When we started to look into Linux and vulnerability management, we assumed it would be a simple task: use these tools to see what’s installed on the computer and correlate that with the the National Vulnerability Database. Easy, right? Well let’s take a deeper look at a package from Ubuntu.

openssl 1.0.1f-1ubuntu2.16

Quick! What version of openssl is this? If you said 1.0.1f, so did we, and we were wrong. The actual version this “equates” to (think Javascript == not ===) is 1.0.1q. So if you’re having a What The… moment, here’s what’s happening. Operating system distro maintainers like to stay in control of things, so when they release a new version of their OS, they create a repository of packages for that release. After the release, as bugs are fixed, the OS can have different ways of handling packages. Some will re-import the package, others will apply patches to their own, existing packages. The point here is that this package is a completely different package from the one you would download from the openssl website. On the server, openssl is now maintained by the distro and is versioned any time there are patches released by the original package maintainer or for commits by the OS developers.

But wait, there’s more! Sometimes, when a bug or CVE comes out, it affects the OS’s package on two different versions of the distro. You may ask yourself, “But what if the fix is different between the different versions?” Well you’ll get this:

Ubuntu 14.04 LTS: mysql-server-5.5 5.5.40-0ubuntu0.14.04.1
Ubuntu 12.04 LTS: mysql-server-5.5 5.5.40-0ubuntu0.12.04.1

What’s happening here is they introduced a new patched version of the package, but threw the version of the distro in the middle of the package version. This only complicates your package comparison further.

So, the next time you’re formulating your plan to patch systems in response to a recently released vulnerability, be aware that some packages require more investigation. If you’re manually comparing packages against CVEs and the contents of NVD to determine if you’re affected, you may need to dive deeper to make sure the update is what you expect on the box!