by Pete Cheslock, Senior Director Operations, Threat Stack
Today we’re pleased to have Franklin Mosley, Senior Application Security Engineer at PagerDuty, contribute to our blog.
Drawing on his extensive experience as an information security professional, Franklin takes a detailed look at the how’s and why’s of integrating security into a DevOps environment, and provides great tips on how you can start making the transition to a DevOps culture at your organization.
I have been in security for many years, so I have heard many of my colleagues complain that developers and operations have little regard for security. But my perspective is a little different: I used to be a software engineer, so I understand the challenges faced in getting software developed and deployed. To that end, I want to share some of my experiences in this post, and hopefully pass along some valuable tips on how to effectively integrate security into your DevOps world.
Software development and deployment has changed a lot over the years. In traditional operations, there were one or two releases per year, they were large scale, and deployment was manual. Today, organizations are releasing much more frequently. Some release every few weeks, and mature organizations are releasing multiple times per day using automated deployment. How are they achieving this? DevOps, of course!
For those who don’t know, the DevOps movement is about CAMS:
- Culture: Working together with minimal barriers, improved communication and collaboration, and increased agility. It’s a cultural transformation!
- Automation: Preventing defects, creating consistency, enabling self service, and helping to eliminate otherwise error-prone manual activity.
- Measurement: Continuous improvement is a hallmark of the movement. How do you know if you are improving without the ability to measure?
- Sharing: Share the tools, discoveries, and lessons learned.
Development and operations teams learned how to work together to achieve improved efficiency, quality, and cost savings in software delivery that resulted in faster delivery of features and functions.
Why was security not involved? Traditional security methods did not fit and were an inhibitor to DevOps agility. In a recent survey conducted by Threat Stack, 68 percent of respondents say that their CEO demands DevOps and security teams not do anything that slows down the business. But organizations are quickly realizing the rising importance of security. High-profile security incidents and data breaches are appearing in the news everyday. Organizations want to maintain the pace of delivery while also reducing their risk of losing revenue and reputation. It’s time to bring on the DevSecOps (or SecDevOps, or Rugged DevOps, or whatever term pleases you)!
What is DevSecOps? Like DevOps, it’s another cultural movement where security is a “shared responsibility” — it’s automated and decisions are made with speed at scale. However, without the integration of proper security thinking and processes, this method of rapid development can introduce security flaws.
It’s important that security activities be aligned with or integrated into software development. If not:
- Quality suffers
- Costs will be higher
- Applications and data are at a higher risk of being breached
I have presented on this topic at several conferences (see DevSecOps: Minimizing Risk, Improving Security and The Security Pro’s Guide to DevSecOps), and one of the most common questions I get is how to successfully integrate security into a DevOps culture. People who recognize and understand the need for security want to know how to help their teams start practicing it. Here are a few tips to help anyone get started:
- Change the security mindset
- Get buy-in from stakeholders
- Enforce security as code
- Be reactive and responsive
Change the Security Mindset
Security teams have long thought that developers were not interested in security. Along those same lines, security teams also thought developers believed that security wasn’t their responsibility. But that’s not quite the case: A 2017 survey from Sonatype shows that 50 percent of developers actually know security is important but don’t have enough time to spend on it. Why? Developers are focused on priorities such as features that enhance the customer experience or improve application performance. They are under pressure to deliver features quickly and on time, and security is seen as a non-functional requirement. So how do we make it a priority?
Learn From Each Other
To make security a higher priority, security and development teams must learn from each other. The security professional needs to walk in the developer’s shoes, and learn how software is made and the issues that are faced. It’s best to work with developers to come up with solutions that allow them to do the right thing when it comes to security.
At PagerDuty, one of our security engineers embedded himself in a development team for a sprint. That helped him learn their tools and processes, and the development team gained insight into some security best practices. With this knowledge, the security team is better able to assess how they can add value and minimize risk to the business without inhibiting the development processes. At the same time, the product team has a better understanding of things to consider to ensure that they are protecting customer data.
Communicate and Collaborate
As mentioned earlier, the DevOps culture includes improved communication and collaboration. But how do you do that? For one, everyone is using the same tools. Our company uses Slack, and each team has a public channel, including our #security team. This creates a forum where people can ask questions about security and also have a running, open dialog that others can follow and learn from. At the same time, the security team is able to join channels owned by other teams, see what they’re working on, and provide advice or help. We also provide weekly office hours where people can drop in to ask questions or seek advice.
At PagerDuty, we believe that the security team are subject matter experts, and it’s our job to make doing the right thing easy. Why is that? With DevOps, we aim to break down silos so security can become a shared responsibility. The agile teams have to own security the same way they have started owning user experience, reliability, and performance. At PagerDuty, our philosophy is that if you code it, you own it. Through open forms of communication and collaboration, we make it easier to take on that security ownership responsibility.
Most developers aren’t trained in secure coding best practices. On top of that, only 1 of the top 36 undergraduate computer science programs in the U.S. made passing a cybersecurity course a graduation requirement. This creates a situation where engineers are not implementing security thinking and awareness into design, coding, and testing. One way to change that is to train them.
Our approach to security training at PagerDuty is to do it in-house. I believe our training has created a mindset where security is more top of mind for our engineers. There is still plenty of learning to do, but security is no longer an afterthought.
Get Buy-In From Stakeholders
Buy-in from executive stakeholders makes security a priority that trickles down to your agile teams and helps foster a culture where it is a priority for everyone in the organization.
I worked at a company where the CEO was conducting a town hall to discuss our new software platform and what it would take to make it successful. He mentioned security of the customer data as one of the pillars of success — it was then that every employee realized they had a personal stake in securing the platform.
To get that buy-in, you can make a case for the importance of securing your software. Many security incidents occur in the news every week, and these can be used to highlight why it should be a priority.
Enforce Security as Code
Automation is a big part of DevOps, and it is no different when integrating security into the process. The manual gating that security traditionally added must be removed to perform security at scale and achieve continuous delivery. Security and compliance should be scripted as available services, integrated into your pipeline, and used to enforce the policies that you have developed.
As an application security engineer, my job is to help the process continue to flow while minimizing risk. Decisions I make should help make the developer’s job easier while enabling the business to make the right decision about security. Whether it’s performing a check of software composition analysis for vulnerable components or performing an automated code review, these tests should be fast and accurate, with the results easy to consume by development teams.
Be Reactive and Responsive
I would love to say that every release is 100 percent secure and that we have 100 percent uptime, but that’s not a reality. Through DevSecOps, you do your best to minimize risk to an acceptable level and ensure that you are deploying a secure, stable, and reliable platform. But we all know that issues will arise. How will you be prepared to handle them?
When I think about how our teams at PagerDuty operate from a SecOps perspective, we follow a model of being reactive and responsive. How quickly can you identify that an event is occurring or that a resource is out of compliance, and how quickly can you respond and remediate the issue? This can further help reduce risk and become a realized cost savings. The learnings from your SecOps should be fed back into your processes as part of your feedback loop. Remember, DevSecOps should be an iterative learning and improvement process.
Starting Your Transformation
I hope you find the tips in this article helpful for successfully transitioning to DevOps culturally while efficiently integrating security into your processes. You can use them as a guideline to encourage communication and collaboration among security, development, and operations teams. Every organization is different, but it is my hope that these tips will serve as a starting point to help you formalize your own processes around bridging the gap between DevOps and security.