Post banner
General 5 Min Read

When We Try to do Right by Security — And Reality Hits Us in the Face!

Recently I was pulled into a sales call to help out with a seemingly simple question from a trial customer. What seemed like a simple protocol misconfiguration turned into a lesson about good intentions, enterprise compatibility, and how — in some cases — we are just insecure by default.

Is this the beginning of an apocalyptic disaster scenario? Definitely not. But it reminded me that, at the end of the day, each organization needs to take responsibility for its own security. On the bright side, the incident underscores the fact that you should proactively decide your organization’s level of risk tolerance and then implement detection that gives you detailed, continuous insight into your workload — with the goal of making sure that your specific definition of secure is being met.

Want the full story? Read on to learn what I discovered while troubleshooting this issue.

To begin, I found that attempts to install our agent package threw an SSL exception similar to the error below.

After reviewing the installation procedure with the customer, I saw that the package was downloaded as expected, but importing the GPG key failed with an SSL connection error. A local test from my laptop showed the file was accessible which, in my mind, confirmed that we were looking at a communication problem. Was an egress rule misconfigured? Perhaps an HTTP proxy in the path was misconfigured or we were hitting a bug on one.

I wanted to know more about the SSL error so — fortunately — I had them use curl to get more information. With the output below, I had an error code to start googling for, so I let the customer know I’d get back to them, and the call continued. Minus one host.

SSL_Error.png

Let me admit to a few mistakes I made while troubleshooting this issue. First, one host succeeded and one did not. I didn’t think to ask what made the two hosts different. Later, I found that the succeeding host was CentOS 7, while the failing host was CentOS 6. Also, I knew the package and key were kept on different sites. Both were accessed via HTTPS. So why was one connection successful and the other a failure?

While researching this issue I remembered a change we had recently made to the Threat Stack platform. We disabled older TLS protocol versions (specifically TLS 1.0) to prevent customers from being able to connect to the platform with vulnerable encryption protocol versions.

So, you say, “You were trying to do the right thing.” Well, yes. We’re in the security business, and we want to follow industry best practices. Protocols like SSLv3 and TLS 1.0 are open to a variety of attacks which can lead to man-in-the-middle attacks or data exposure. This is also where we host the repository signing keys. Hosting the repository GPG key on an alternate site, a practice not commonly performed, helps prevent a site breach from uploading both altered packages and an altered key to go with them.

Look at all our good intentions! We were protecting our customers! We were protecting them from years-old vulnerabilities! We were restricting them to use encryption protocols that have been around for nearly a decade! Could these good intentions and this customer issue be related?

Eventually I came across an interesting bug as I tried to figure out what was wrong:

curl supports TLS 1.0 as the highest SSL/TLS version

“Wait, what? Is this serious? The version of curl on RHEL 6 (and therefore CentOS 6) only supports up to a 15 year old encryption protocol that has been shown to be insecure? That can’t be …” Oh yes, it can. And the issue is not going to be fixed! The ticket doesn’t have all the details, so let’s discuss the issue and the response.

The ticket was opened up against RHEL 6.5 with the issue that curl did not support encryption past TLS 1.0. The reporter mentions the prevalence of TLS 1.1 and 1.2 and the oblique reference to the vulnerabilities in previous protocol versions. By RHEL 6.7, a fix was shipped to support later protocol versions. Here’s the changelog for the next package release:

* Mon Dec 1 12:00:00 2014 Kamil Dudka <[email protected]> 7.19.7-43

– fix manpage typos found using aspell (#1011101)

– fix comments about loading CA certs with NSS in man pages (#1011083)

– fix handling of DNS cache timeout while a transfer is in progress (#835898)

– eliminate unnecessary inotify events on upload via file protocol (#883002)

– use correct socket type in the examples (#997185)

– do not crash if MD5 fingerprint is not provided by libssh2 (#1008178)

– fix SIGSEGV of curl –retry when network is down (#1009455)

– allow to use TLS 1.1 and TLS 1.2 (#1012136)

– docs: update the links to cipher-suites supported by NSS (#1104160)

– allow to use ECC ciphers if NSS implements them (#1058767)

– make curl –trace-time print correct time (#1120196)

– let tool call PR_Cleanup() on exit if NSPR is used (#1146528)

– ignore CURLOPT_FORBID_REUSE during NTLM HTTP auth (#1154747)

– allow to enable/disable new AES cipher-suites (#1156422)

– include response headers added by proxy in CURLINFO_HEADER_SIZE (#1161163)

– disable libcurl-level downgrade to SSLv3 (#1154059)

But why was the package installation still failing? The ticket goes on to say:

As far as I know, there is currently no plan to make curl use TLS 1.2 by default on RHEL-6. You can either use the –tlsv1 option of curl to negotiate the highest version of TLS supported by both client and server, or update to RHEL-7, where this behavior is used by default.

While the ability is there to support secure protocols with curl, they are left disabled by default. No reason is given in that ticket, but one can assume it was to preserve application compatibility on RHEL 6. That was the justification given here for RHEL 7, which ultimately was updated: https://bugzilla.redhat.com/show_bug.cgi?id=1170339

What is the justification for curl doing this?

Compatibility. We prefer not to change the default behavior in Enterprise Linux while we allow applications to use new features if explicitly requested. This is not specific to libcurl.

Lessons Learned

So where does this leave us and what can we learn? First as a user, you can apply all the security updates from your vendor — and still be vulnerable. How many sites are using RHEL 6 and curl or curl-based utilities (such as the YUM package manager) to perform operations in their environment? How many sites blindly use SSL/TLS assuming they’re protected?

But is the sky falling with this issue? Probably not. Exploits against TLS 1.0 are difficult to perform, and there is much more low hanging fruit out there. But I say “probably not” for an important reason: You need to decide your own risk tolerance and also know your compliance needs. A non-issue for one organization could be a critical issue for another. Additionally, different risk tolerances between your organization and your vendor may leave unexpected gaps in your infrastructure.

This is where defense in depth plays a crucial role in securing your infrastructure. In addition to performing your routine patch management, you should always be thinking about and monitoring for detection to ensure that your own definition of secure is being met.