I’ve spent most of my career in Operations, and the last 5 years at various organizations advocating and instilling DevOps principles in the teams I work with. One thing I’ve noticed is that most companies value speed over security, which has traditionally been a blocker in delivering software.
Recently, however, with more and more breaches and vulnerabilities reported (Shellshock and Heartbleed to name a just few), I’ve changed my tune. I’m not going to say I’ve become paranoid, but one of the reasons I’ve joined Threat Stack is because I believe how important it is that security gets integrated into the operations process.
Based on deep experience building out infrastructure while moving fast, I detail in this post the top four ways to balance DevOps workflows while maintaining a pragmatic view on security.
How to effectively balance DevOps workflows and cloud security practices
1. Recognize that the tools which enable DevOps to work so well also introduce new threats and attack surfaces
There are a lot of new applications that have entered into the modern-day operations toolbelt: Vagrant, Packer, Docker, Chef, Puppet, Salt, Ansible, etc, etc. The problem that many of these tools solve is the ability for engineers to manage more systems with more control and consistency than ever before. Since operations engineers are now representing their infrastructure as code, they are able to deploy changes with a much higher velocity and much less risk than ever before. Additionally, since their infrastructure is now represented in source control, they can use software engineering methodologies to test their code, and are able to do so far earlier in the development process.
Now with tools like Docker and OSv, and the maturity around orchestrating containers and processes with tools like Kubernetes, Mesos and others, operations engineers are maintaining a platform that enables their software engineers to ship changes quickly and effectively. The wall that had existed between Engineering and Operations has largely been broken down with both sides embracing change and pushing updates frequently. But now we’re faced with a new wall, one that exists between Security and the rest of the technical organization. Fortunately, there are collaborative ways to overcome this wall, as I’ll go on to explain.
2. Mitigate risk while still moving fast
It’s the classic “DevOps” conversation over again: Developers want frequent change and/or are being pushed forward for change by their Product teams, yet Operations teams want stability, and any changes to their systems could introduce inherent instability. Now, both Dev and Ops teams are enabling each other to ship effectively using many of these new tools to assist them. Yet Security teams, for the most part, have been left behind.
What I hope to see over the next few years is Security teams using and creating new tools that enable them to insert security and risk control into the same pipelines that Engineering and Operations are using to deploy their changes. (This is a big problem that I’m excited Threat Stack is solving for Dev, Ops and Security teams). For example, if Ops teams are using Chef to continually update their systems at scale, Security should be writing cookbooks that setup and enforce security policies. If Engineering is using Jenkins to continuously integrate their code, Security teams should add additional tooling into the build pipeline to monitor for risk and threats.
3. Let developers and others have safe access to production
One ancillary benefit to companies that have implemented an effective DevOps culture change has been increasing ownership across all teams. A big part of increasing ownership that I’ve specifically used in my companies has been to put the engineers on call for the specific applications they own. For example, the web team has a pager rotation and gets alerts when there are issues with the web servers, and the search team gets alerts when search clusters are OOMing. However, this can present a frightening reality to Security teams who now have to deal with both frequent changes to the code and an endless group of people who have access to systems.
I’ve taken a “trust but verify” methodology when it comes to access control, and in the past have implemented tools including auditd, OSSEC, centralized logging, etc. to ensure compliance. But these tools are cumbersome to setup and configure, and with tools like auditd, have a significant performance hit to your systems. You never want anything like that to get in the way of progress OR performance, which is why carefully increasing ownership works well among teams.
4. Build a system that will allow you to regularly push to production with security checks in place
Odds are that your company already has some sort of CI tool in place, such as Jenkins. These same CI tools that Engineering has been using to continuously integrate their feature branches are the same tools Operations are probably using to manage and deploy their code — whether or not this is the ideal solution. Security teams should reach across the aisle and work to integrate their security tooling directly into the automation pipelines to allow for those quick feedback loops that Engineering and Operations have become used to.
By building in automation, taking full advantage of virtualized cloud infrastructure, and embracing a workflow that empowers developers to push code continuously, this enables companies to move fast when it comes to deploying and managing apps, all with less overhead.
As DevOps has evolved into a mainstream philosophy, we’ve seen more and more DevOps conversations floating towards security-related topics as the next logical part of the equation. With an increasing number of conversations like these happening, it’s more important than ever that there is a unified understanding of the ways to balance new DevOps workflows with evolving cloud security practices. My entire team and I at Threat Stack are excited to be solving such an important problem. To see how we’re enabling teams to integrate these workflows, check out threatstack.com for a demo.