Why Did We Need to Invent DevSecOps?

While the term “DevSecOps” has started to come up more often recently, we’re still wrapping our heads around “DevOps” to answer questions such as “How do I implement DevOps?”, “Where do I find DevOps engineers?”, and “What does DevOps even mean because I just asked 6 people and got 8 different answers?”

And now this new term “DevSecOps” comes along.

While “DevSecOps” is coming into use much more frequently, it still doesn’t have a precise meaning. To help explain why the term is surfacing, this post explores how Development and Operations evolved into DevOps (leaving Security behind). It goes on to explain that we’re now at a point in cloud evolution where Security can no longer be ignored and that, going ahead, the relationship between Development and Operations must be expanded to include Security. And hence: DevSecOps.

The goal of DevOps is to rethink our software delivery model to incorporate the needs and viewpoints of both Development and Operations groups into the planning, development, and delivery process. DevSecOps is just like that! Except we’ve bolted Security onto DevOps as if it’s an afterthought. And the truth is, we have forgotten about Security to some degree. The fact that we’ve had to invent a new word to describe what many of us have been doing for years, but now with Security added, is evidence of that.

The Bad Old Days of Service Delivery

So why did we have to invent DevSecOps?

Let’s start in the bad old days of service delivery. This was the world of silos and asynchronous processes. Let’s forget about Development for a moment and just focus on the Operations silo which was, in fact, its own collection of silos. I could easily provision a VM to run a new service. (That was actually my job as a sysadmin.) Shoot, I need a little bit of storage for that host. Time for a ticket to the Storage folks. Wait, it needs to talk to that database over there too. Okay, let’s change the VLAN the VM is in, but before I can do that I need to get an IP from Network Engineering.

Sooner or later — once everyone got back to me — I got a host up and running. And if I didn’t have any strange requirements, I probably didn’t run afoul of Security unless sometime down the road the host popped up onto one of their scans. Notice a difference between Storage and Networking compared to Security? I never wanted to talk to any of them if I could avoid it since they slowed me down. But at least Storage and Networking provided me with a service to consume and use. Security just told me I couldn’t do something or had to redo my work.

The Explosion of the Cloud and DevOps

Then some things changed with the explosion of the Cloud and DevOps. Remember Development, which we forgot about earlier? They looked at the Cloud and wondered what was so hard about provisioning a new host. It was just a few button clicks, and there was an API so they could write a script to make the provisioning even easier. As businesses migrated, or in many cases started in the Cloud, Operations was forced to explain what value they brought to the organization. The Cloud-native organizations, in particular, started to see the value of Operations engineers as they started to scale. Operations started as work that was distributed among developers or was dumped on the developer who knew just a little more than everyone else. This led to frustration in developers since it got in the way of doing the sort of development they wanted to be doing. Who better to do this work than professionals who enjoyed doing it — in particular, those Operations engineers who saw the trends and adapted. These were Operations engineers who didn’t see themselves as the people who doled out resources or who stopped developers from deploying services that would wake the on-call ops engineer. Instead they saw themselves as the folks who built the tools so developers could allocate resources on their own and worked with them to ensure that their service didn’t wake the owning engineer in the middle of the night.

With this jump to the Cloud, certain skills were no longer necessary. Didn’t need a storage engineer anymore. No need to design, build, and maintain a storage array network. Instead, just create a bucket and toss your data in there. Sure it wasn’t necessarily as fast or reliable, but those limitations were designed around. Didn’t even really need a network engineer either. Up until a few years ago, Cloud networks were essentially flat with the equivalent of a firewall stopping or allowing traffic. Even today’s more sophisticated Cloud networking is basic networking skills.

What Happened to Security in This Transition?

But what happened to Security in this transition? The Cloud brings new threats that needed solving and no ready replacements as it did for network and storage engineering. Security was left out of this transition for several reasons. They didn’t bring immediately necessary skills, too many stuck to traditional patterns instead of evolving; and again, they were seen as slowing down the service delivery cycle that Development and Operations had worked so hard to fine tune.

The Operations engineers who made the jump to DevOps and Cloud early on brought with them the skills to build the tools that transformed them from gatekeepers to enablers, their operational knowledge and patterns, and a desire to adapt those patterns to meet the needs of developers. Operations was long the domain of engineers who didn’t want to code that much, if at all. The first Operations engineers to make the jump were the minority with reasonable coding skills. Those in Security with those skills were a minority of a minority in Operations. The end result was small numbers of Security engineers being able to transition and be perceived as immediately valuable. To make matters worse, their work provided no consumable service. Security’s work wouldn’t allow a developer to provision security like they could hosts or storage. The abstract value they provided worked against them. Security knowledge that did make the jump was typically rudimentary, just as the storage and networking knowledge was.

Next there was unwillingness to adapt. “You can’t secure data in the Cloud. You don’t know who else has access to it on someone else’s system. There’s no way we can build enough defenses to keep people out. You’re just inviting a breach.” That mentality was no different from that of the Operations engineers who thought you couldn’t engineer reliable systems in the unreliable Cloud. The Operations engineers who embraced the challenge and adapted to build reliable systems were a minority at odds with the majority of their peers. The same goes for Security engineers who had to argue against the idea that data is no longer to be trusted once it leaves your hands. Sure you could ship a tape to a secret mountain lair by giving it to some guy in a van, BUT THEIR JOB IS TO SECURE YOUR DATA! Your Cloud provider wasn’t making any claim like that. Even worse, very few Security engineers wanted to advocate untested ideas that could result in having to explain a potentially costly data breach.

Finally, Operations and Developers were starting to make this whole DevOps thing work and much of it was done with minimal security oversight. Once these Cloud-native organizations started scaling to the point of needing dedicated security, a gap in mentality had arisen. Operations had learned to ship services very fast. A developer could ask to develop their own service against a new database, Operations could quickly return a usable instance that a developer could develop against, and Operations would continue their work of ensuring metrics, monitoring, alerting, tuning, and reliability. When Developers and Operations were satisfied, they could also ship into production. Sure we (hopefully) did rudimentary security checks like using SSL/TLS… Oh, this service doesn’t support those protocols. Well, there’s nothing we can do about that, and there’s no alternative so let’s just ship it. The username and password the developer’s service uses is stored in config management… And we’ll get to implementing encrypted secrets sooner or later. (At least the secrets are not embedded in the code!)

The Need to Integrate Security Re-Emerges

Now Security comes along telling us we need to do things differently. We need to go through security reviews of what gets shipped. We need to drop what we’re working on to implement a compliance requirement. This is a world many Operations engineers were happy to leave behind. Security was battling not just developers, but Operations teams that had become accustomed to a new style and pace of work.

So why did we need to invent DevSecOps? Cloud usage has grown significantly in the past few years. Cloud-native organizations have successfully scaled to the point where growth isn’t their only objective. Their success brings with it the need to protect their massive valuations or regulatory compliance. Organizations transitioning to the cloud have started adjusting their development practices and reorganizing their operations teams to achieve faster delivery, modeling themselves after the lessons learned from those that made the Cloud and DevOps transitions ahead of them.

But where is Security in all this? Given the amount of business that is in the Cloud or is transitioning to the Cloud, there’s a real sense of urgency surrounding Security. And it’s time for a discussion of where Security fits into this world when the teams around them are changing. A number of key questions need to be answered including the following:

  • First, what has Development and Operations been doing wrong as it  prioritized the velocity of service delivery? How many services have gone out unknowingly misconfigured? How many times has opening up security group rules been the quick and easy answer to letting services communicate (overriding the reasons they were segregated in the first place)?
  • Second, what do we need to readdress in terms of the way that Security performs its integral functions? Security teams need to learn from the Operations teams that transitioned before them what skills, techniques, and mindsets are effective in becoming an active part of service delivery instead of an organization to be avoided. They need to learn how Operations teams transitioned from gatekeepers to allowing developers to safely provision their own resources. How did they learn to transition from a relationship with Development that was, at times, adversarial, to a collaborative one?

We invented the term DevSecOps to bring Security back into the service delivery equation and as a way of answering the critical questions just asked.