How to Create a Security-Minded DevOps Organization: Three Best Practices

You’re a week into your new job and a colleague shouts out across the room before a big deployment: “Hey John, you’ve got security covered, right?” You rush over to your good friend Google for a few quick ideas on implementing security best practices into DevOps and timidly shake your head “yes” at your colleague.

This might sound pretty familiar to you. In fact, it’s how I felt at previous companies and even upon joining Threat Stack, until I was immersed into the world of security. With everything moving faster as workloads migrate to the cloud, everyone wants to avoid being the next big breach victim, but as I’ve learned here at Threat Stack, DevOps and security need to work together in order to do this.

In today’s culture of continuous deployment and delivery, it’s not just about moving fast. You need to simultaneously ensure that your organization is secure and compliant. But how do you do that if you don’t have a dedicated security person on staff or don’t know security yourself?

I teamed up with Carmen Andoh, an infrastructure engineer at Travis CI, to host a webinar this week to discuss three practical ways DevOps teams can run fast and scale securely. If you missed this webinar, you can view the entire replay below.


We also put together the following overview of what we covered so you can quickly review the highlights.

1. Start Somewhere (security doesn’t have to be hard)

Oftentimes, DevOps teams don’t feel empowered to take on security because they don’t know the jargon or have the know-how to implement it. How to classify security events between a Severity 1, 2, and 3 alert, for example, can be baffling.

Modern-day security isn’t a superpower that only certain people can wield, and jargon shouldn’t be a roadblock to integrating security into DevOps anymore. I have actually learned a lot about security just by using Threat Stack’s platform since I started working here. And I haven’t needed to learn a ton of jargon to throw my hat in the ring.

For example, one thing I did early on was create a new CloudTrail rule in Threat Stack that said if a user creates a server outside an auto-scaling group, I want that to bubble up as an alert. It didn’t have to be a “wake me up in the middle of the night” alert, but it would be a signal that something bad might be happening. Really, all I was trying to do was gain more visibility into what was going on, so instead of focusing on the definitions of severity levels, I just defined what I needed to know, when — and Threat Stack did the rest for me.

There are other “low-hanging” security tasks you can do that can have a big impact. One is as simple as turning on nightly software updates for your servers. While advanced persistent threats and zero-day attacks get a lot of buzz, research shows that it’s actually good old unpatched software that results in the most breaches and attacks.

Vulnerability management can automate a lot of this for you, keeping systems up-to-date, tracking what’s going on across your environment, and identifying vulnerabilities in workloads based on common vulnerabilities and exposure (CVEs). Threat Stack’s vulnerability management features are some of my favorite aspects of the platform (and they’re easy to use as an Ops Engineer), because they give me high-level knowledge of what’s going on. I can see where possible attack vectors are hiding in our servers and quickly determine how to respond.

When it comes to security, if you try to boil the ocean and do too much at once, it won’t end well. Instead, start by scoping out how you can gain visibility into who did what and when. This way, you can mitigate a lot of risk without having to learn a ton of jargon or complex security concepts.

2. Trust, But Verify

In a continuous delivery world, many people typically have access to production, and more systems are hooked into each other to get things done faster. So how do you verify that activity is legitimate vs. malicious, especially when things are moving fast? Many companies are turning to the new collaboration model, ChatOps. Gaining a lot of popularity over the past several years, ChatOps streamlines events from all your tools, alerts, and processes into a single chat systems like Slack, Hipchat, or VictorOps, turning it into a new terminal for infrastructure communication. This can have huge benefits for security.

At Travis CI, for example, they have a dedicated incident response channel on Slack that pulls in communications from all their tools so that everyone can monitor company operations. They then leveraged Threat Stack’s Slack integration by creating a channel to pull in security alerts. Now, anytime something suspicious comes up in the incident response channel, they can quickly hop into the Threat Stack channel to verify what happened.

At Threat Stack, we have a similar process in Slack. When a developer is about to do something that they know will fire an alert, they pop into the channel to let everyone know, so when the alert comes in, we know we can ignore it. But if no one says anything before an alert comes in, or an alert happens at an odd time of day, we can assume that something might be going on that’s worth investigating.

3. Everyone is Responsible for Security

Threat Stack and Travis CI both embed security into our company cultures. Security is owned by everyone, not just the security folks. Even companies that don’t have security teams can benefit from this approach, allowing security to become a possibility even at small companies with limited resources. The concept is simple: If everyone has a security mindset, then it will be embedded in everything we do.

When you have the right tools, such as Threat Stack and Travis CI, you gain visibility across your environment and are empowered to do security work — no crazy jargon required. By leveling the playing ground with tools that help you jump right into a security mindset, security almost takes care of itself.

A Final Word

For those of you who attended this week’s webinar with Carmen and me, thanks very much for joining us! For anyone who missed it, have a look at the full webinar (embedded earlier in this post).