DevSecOpsSecurity Research & Strategy

20 DevSecOps Pros Reveal the Most Important Considerations in Building a DevSecOps Pipeline

In a recent Threat Stack report, 44 percent of DevOps professionals we surveyed said that when it comes to security-related issues, they’d have to rely on someone else. Even if DevOps pros had the time to dedicate to security issues, many developers lack the expertise needed to improve the security of their applications. What’s more, security proves to be a significant roadblock in application development: 40 percent of those surveyed at this year’s RSA conference reported that the impact on agility and speed of application development and deployment is their most significant roadblock when it comes to implementing application security programs.

One solution is to introduce security earlier in the development process, but that’s often easier said than done. Threat Stack is purpose-built for Operations and Security teams running in the cloud, offering a security platform that’s intuitive for Ops teams so they can take ownership of security as well as complete visibility so you can take prompt action on suspicious behavior. And, Threat Stack’s Cloud Security Platform® now includes Application Security Monitoring at no additional cost to help you address common DevSecOps challenges —  all without slowing down your DevOps processes and workflows.

As more companies look to integrate security into the DevOps process, following best practices is key so DevSecOps becomes a benefit rather than a hindrance to your DevOps team’s productivity. To learn more about key considerations and best practices for building a DevSecOps pipeline, we reached out to a panel of DevSecOps pros and asked them to answer this question:

“What is the most important consideration in building a DevSecOps pipeline?”

Meet Our Panel of DevSecOps Pros:

Keep reading to learn what our experts had to say.

Disclaimer: The views and opinions expressed in this post are those of the authors and do not necessarily reflect the policies or positions of Threat Stack, Inc.

 Dmitry Sotnikov

@42crunch

Dmitry Sotnikov is Vice President of Cloud Platform at 42Crunch. 42Crunch is the only enterprise-grade API Security platform that addresses the development, testing, and deployment security requirements of an API infrastructure. 

“DevOps is often viewed as the ultimate way to be agile…”

Release quickly, get feedback, fail fast, iterate continuously — and you will get ahead of your competition and have the best product in the market.

This is viewed as a much better approach than spending weeks or months doing a proper design and then executing according to it only to find that your product is not what the market needs and your competition has already eaten your lunch.

In reality, viewing these as opposites is leading to disastrous results when it comes to security. Engineers go all-in into microservices architectures, the APIs of these services are viewed as an implementation detail, the overall product becomes popular and gets traction, and then attackers find flaws in API design and implementation and take advantage of the application and customer data.

LandMark White (LMW) can serve as an example of things going terribly wrong. This was the number one property valuer in Australia, and all major banks were using LMW APIs to get property estimates for mortgages and other transactions. Unfortunately, one of the backend services was not properly secured, attackers found a way to get in, and a huge volume of transaction information got downloaded and leaked to the internet. As a result, all major banks stopped using the service, the company was pulled from the stock market, and the CEO and the leadership team had to step down.

In a less dramatic case, India’s number one local search service, JustDial, ended up having personal details on more than 100 million of its users leaking to the internet. As their app team followed DevOps processes and iterated, one of the APIs became unused. So, while security of the app itself and newer APIs improved, no one was paying attention to the older API still being available and having access to an extremely sensitive database with customer data. When attackers found and exploited that API, it took the company time to figure out what was really going on.

The solution is to stop treating DevOps as the opposite of careful security design. While fast iteration of features in DevOps is indeed helping products evolve faster, it is imperative that security gets enforced on each and every iteration. Iterate on scope but not on security.

It is totally fine to start with just one microservice with a simple method or two — but make sure that this simple API is bulletproof: carefully designed to do only what it is supposed to do, accept only the payloads it is supposed to accept, and respond only with the data your users need, with proper authentication and authorization.

Get that security enforced all along the CI/CD pipeline: from static analysis of the API contract, to API implementation, to runtime protection with a gateway or API firewall. Then iterate on the next methods for that microservice or other microservices.

Thus, DevOps becomes DevSecOps, and security is not compromised while the company successfully iterates on the project.

 Javed Shah

@ForgeRock

Javed Shah is the Director of Product Management for Cloud and DevOps at ForgeRock.

“The key to successful DevSecOps is automation of security controls inside the DevOps pipeline…”

Ten principles to follow for DevSecOps implementation:

  1. Principle of least privilege for all services that process (read, write, or update) data.
  2. Enforcing tight access security for API endpoints.
  3. Running SAST (static application security testing) tools as part of the nightly build process and running DAST (dynamic application security testing) tools to identify security defects in running containers.
  4. Scanning any pre-built container images for known security vulnerabilities as they are pulled into the build pipeline.
  5. Automated tests for security capabilities wired into the acceptance test process. These automated tests include input validation as well as authentication and authorization enforcement.
  6. Isolation of containers from one another, avoiding any dependencies and keeping them entirely stateless to eliminate high-value targets for attackers.
  7. Automated security updates, such as patches for known vulnerabilities, by means of the DevOps pipeline with an audit log.
  8. Reduce attack surface by using a secure API gateway that enforce fine-grained and scope-grained access to sensitive API endpoints.
  9. Automated service configuration management, allowing for compliance with security policies and the elimination of manual errors.
  10. Continuous monitoring, audit, and remediation of security defects across the application lifecycle.

Also, firewalls should continue to defend in depth by isolating services. Intrusion detection is a lot harder using containers, so looking at network behavior helps detect abnormal traffic patterns. If possible, security tooling should be a gate to deployments (applies to SAST and DAST).

However, all this automated flow should still be validated by external pen tests to make sure automation covers all aspects. Additionally, Incident Response plans should be created and practiced for all new environments to ensure they have the capability to preserve evidence to aid in investigations and staff knows how to execute the plan, either themselves or who to outsource it to.

One of the great benefits of DevSecOps is its democratizing effect on security, but there’s still a need for security subject matter experts. Blended teams allow everyone to contribute to security, but there are some topics — cryptography, secrets management, and secure coding, to name just a few — that are difficult to master and easy to foul up. Developers should pay special attention to the handling of unencrypted and untokenized sensitive data (secrets) within the code, both during code design and peer reviews — even scanning code before checking in changes using standard security scanners available as IDE-plugins. Part of the art of assembling an effective DevSecOps culture involves building the right mixture of skills on your team.

The biggest thing that not only developers need to deal with is the changes that are happening and in some cases the speed of those changes. This might include processes, tooling, and even personal knowledge. Part of this shift left is that everyone now has to be at least familiar with all aspects, this might include network security, application security, IT, performance, etc. Often, this can be overwhelming to a developer and leads to cutting corners.

 Joseph Feiman

@whitehatsec

Joseph Feiman, PhD, is the Chief Strategy Officer at WhiteHat Security. Feiman is responsible for the company’s overarching business strategy and vision. Previously, Feiman worked for Gartner as Research Vice President and Fellow. During his tenure at Gartner, he served as a trusted resource for security executives and professionals across the globe, co-founding the application security market category.

“The truth is that organizations knowingly release insecure code all the time…”

It’s time to stop blaming hackers and focus more on preventing the release of insecure software. Properly secured applications are a win in the long term, both for companies and society. The most important consideration in building a DevSecOps pipeline is ensuring comprehensive security testing. This can be done in three steps:

  1. Static application security testing (SAST), which analyzes application code and detects vulnerabilities.
  2. Dynamic application security testing (DAST), which analyzes applications and detects vulnerabilities at run time. It launches simulated attacks and analyzes the reaction to determine if there is a vulnerability.
  3. Software composition analysis (SCA), which analyzes applications for third parties for open-source software. It detects illegal, dangerous, or outdated code.

 Mark Runyon

@aspprogrammer

Mark Runyon works as a principal consultant for Improving Atlanta. He focuses on systems architecture and development of enterprise web and mobile applications. As a full stack programmer, he primarily works in the Microsoft stack, leveraging cloud technologies in Azure and AWS.

“For a DevSecOps CI/CD pipeline, you are looking at two main areas of security testing…”

SAST (static analysis security testing) looks for things like SQL injections or cross-site scripting. This approach focuses on the application from the point of its code and isn’t concerned with execution. These checks can be run against each feature that is checked in, so issues tend to come to light quickly. Vulnerabilities in open source code included in the project would be exposed under SAST.

DAST (dynamic analysis security testing) targets the application as a hacker would. It scans the executing application to root out vulnerabilities through penetration testing. DAST testing comes in later in the development life cycle.

Effective DevSecOps takes into account both when developing CI/CD pipelines and may use multiple toolsets to identify each type.

 Markku Rossi

@SSH

Markku Rossi brings 25 years of software engineering and architecture experience to SSH.com. He is responsible for R&D and directs the company’s technology strategy. Markku has extensive knowledge and experience with SSH Communications Security products, having served the company from 1998 through 2005 as a Chief Engineer and a major contributor to the SSH software architecture.

“Avoid security risks while making life easier for developers…”

Developers are constantly transitioning between IT resources, which may live on-premise or in hybrid IT environments that combine multiple cloud infrastructures. They need to be able to do this with speed, agility, and elasticity.

The last thing they want are processes that slow them down. Unfortunately, secure access today often means doing just that — slowing down and stopping to manually input credentials across environments. Developers might also share credentials with each other to try and speed up work, but that creates yet another security risk.

Even privileged access management (PAM) solutions, which are supposed to simplify and centralize IT credential management, are traditionally too bulky, complicated or non-intuitive for modern workflows. Rather than deal with all of the typical secure access challenges, some developers will often bypass PAM solutions altogether, creating keys that exist outside of the PAM solution, meaning they are untracked, unmanaged, and a major security risk.

The use of ephemeral certificates is a valuable alternative. They eliminate manual credential inputs by providing just-in-time access, because access credentials are automatically created on-demand and require no installation, configuration, or updating. In other words, they free up developers to focus exclusively on the work they do best.

They are inherently more secure because they are temporary — only existing for the period of time during which access is required — and automatically expire as soon as you are done using them. As a result, there are no credentials to lose, misplace, or steal, leaving fewer old credentials lying around for someone to take advantage of down the road. On top of that, ephemeral certificates don’t leave user-specific access credentials stored on target servers, so if someone does get access to your target server, they won’t be able to find any credentials that could be used to hop to other systems.

 Ted Shorter

@Keyfactor

Ted Shorter is the CTO of Keyfactor.

“Developers specialize in code, not security, hence the rise of DevSecOps and core best practice…”

Code signing is a fundamental consideration that must be bolted throughout the DevSecOps pipeline. Recent research underscores the need for tighter code controls, pegging the average cost of the misuse of code signing keys and certificates at $15 million.

Security and PKI teams would prefer to isolate and lock down private keys, but developers need quick access to sign code and push it to production. The biggest problem is how to implement safeguards to prevent misuse of keys and certificates without impeding the productivity of the team’s developers.

Here are four fundamental steps every DevSecOps organization can take to ensure that code signing is baked into their pipeline:

  1. Get a complete inventory of your landscape. Digital certificates and keys used for code signing are high-value assets, yet 71 percent of organizations don’t know exactly how many they have. Start first by taking an inventory of how many code signing keys you have, where they live, and how they are stored.
  2. Store private keys in a certified HSM. Hardware security modules (HSM) are the most effective way to ensure that your private keys remain under your control. Keys can be stored or generated inside the HSM and used to sign code anywhere without ever having to leave the hardware.
  3. Centralize management of keys and certificates. Local code signing creates silos in security and increases costs. A centralized, server-side solution can simplify administration, improve security, and eliminate the need to have a separate code signing certificate for every developer or build stream.
  4. Choose a flexible solution. Choose a solution that is easy to deploy and scale as demand for code signing grows. Cloud HSM services offer all the benefits of hardware-level security, but without the upfront expense, manual set-up, and ongoing maintenance of dedicated hardware HSMs.

 Morey Haber

@MoreyHaber

With more than 20 years of IT industry experience, and author of Privileged Attack Vectors and Asset Attack Vectors, Mr. Haber joined BeyondTrust in 2012 and currently oversees the vision for BeyondTrust technology encompassing privileged access management, remote access, and vulnerability management solutions, and BeyondTrust’s own internal information security strategies.

“One thing to consider is helping teams get past the technical, cultural, and emotional challenges by…”

Explaining the benefits of DevSecOps and reinforcing that no one is being punished if they lose access, credentials, permissions, or authority to assets, applications, or resources they managed in the past. This is an education problem. Technical cultures need an explanation as to why things are changing and why the changes will benefit everyone — from security to operations.

While this may sound a bit tongue-in-cheek, I like to use the example from Charlie and the Chocolate Factory (the one with Johnny Depp). Charlie’s grandfather worked in a toothpaste factory for low wages screwing the top on toothpaste tubes. He was replaced by a robot and ultimately fired. When the robot broke, someone needed to fix it. Charlie’s grandfather took the new job to fix it for higher wages and learned a new skill set.

DevSecOps is the same (no, you’re not going to be fired!). Teaching someone to join a Windows host to a domain, for example, is easy within the UI. But teaching them to write a PowerShell script, automate it, and maintain it for new versions of Windows is harder, more challenging, and should command a higher salary. Admins will no longer need administrator rights to a system but must learn how to automate the steps based on business goals.

Therefore, educate teams as to why you are changing, who can step up to the new requirements, and help educate them to grow and re-invent themselves to the new challenges DevSecOps will mandate. DevSecOps will require maintenance and development in a secure fashion, just using a different paradigm then what teams are used to today.

 Dawn Parzych

@dparzych

Dawn is a Developer Advocate at LaunchDarkly, where she uses her storytelling prowess to write and speak about the intersection of technology and psychology. She enjoys helping people be more successful at work and at life. She makes technical information accessible avoiding buzzwords and jargon whenever possible.

“Look at where to introduce security checks into the existing pipeline and processes without breaking what currently exists…”

Part of DevOps is the notion of iterating and continuously improving. Security should be introduced in an iterative manner. Look for small steps and changes that can be made. Introduce those changes and examine what worked and what didn’t work, make changes, and move forward. Getting some quick wins and improvements in processes in place will reduce friction when building a DevSecOps pipeline.

 Elliott Spira

@GorillaStack

Elliott Spira is CTO of GorillaStack, a cloud automation company that specializes in Security & Governance automation on the cloud.

“When implementing DevSecOps, it is important to remember that…”

It is a broad concept, or combination of themes rather than a well-defined practice. Before implementing your pipeline, it is worth thinking about which of the key themes in DevSecOps we are satisfying by implementing a pipeline:

  1. Empower Engineering Teams: Dev and Ops are responsible for creating secure applications and infrastructure.
  2. Shift Left: Start security in development, rather than retrospectively in staging or production.
  3. Security as Code: Use tools and automation to do security in a declarative fashion, and version control our checks and security practices in lockstep with the code.
  4. Continuous Security: Just like unit tests, or code styling, we should be asserting our security expectations on each version of code considered for building and deployment.

Thinking about DevSecOps as just another key part of the Software Development Lifecycle (SDLC), it is easier to get started. Start small and improve incrementally. Like adding unit tests to projects, even if the test coverage isn’t perfect, each unit test is valuable in and of itself.

Take this same approach to your DevSecOps lifecycle and you should be able to get some momentum going.

 Sridhar Jayaraman

@qentelli

Sridhar Jayaraman is the VP of Engineering at Qentelli. He has over 18 years of experience in IT Delivery, Consulting, and Governance across multiple companies and geographies. Passionate about technology and its ability to change peoples’ lives for the better, Sridhar believes in devising and delivering solutions that provide true business value.

“Before you build the DevSecOps pipeline, conduct a threat risk analysis to…”

Determine the type and level of threats, impact of an attacker gaining highest privileges on the application/system running the application, and determine the type of data that the application stores is one of the most important considerations.

You could have various tools in the pipeline, but if the tests that are run don’t address the threat profile, your organization may get a false sense of security when all the tests pass. Further, the threat profile can be helpful in identifying the right tools to run the tests.

An equally important consideration is how the tools and tests are run in the pipeline — if the tests either return too many false positives or if it takes too much time to run those tests every time there is a build, the team may slowly lose confidence and remove the tests from the pipeline “temporarily,” which then becomes regular.

 Doug Barbin

@schellmanco

Doug Barbin is the Principal and Cybersecurity and Emerging Technologies Practice Leader of Schellman & Company, LLC, a global independent security and privacy compliance assessor.

“There needs to be traceability in an effective DevSecOps process…”

Actions taken by DevOps personnel need to be logged, especially those with justifiably higher levels of access that may bend traditional separation of duties definitions. Additionally, the procedures followed and tools utilized to perform security testing also need to generate trails to show auditors, regulators, and customers that controls are operating.

DevOps is reality; it has been for some time, and companies are utilizing the delivery model for the right reason. However, as an auditor, we frequently come in to see the commercial benefits while a software or cloud provider struggles to prove that the Sec in DevSecOps is really in place.

 Alan Zucker

Alan Zucker is the Founder of Project Management Essentials. He is an instructor and advisory consultant helping organizations with their Agile and DevOps practices. He played a leadership role in an Agile/DevOps transformation at a Fortune 50 financial services company.

“DevSecOps is the next iteration in developing an environment and culture of continuous delivery of value…”

Just as with DevOps, one of the primary first steps is breaking down the organizational and cultural silos that separate information and infrastructure security from the development and technology operations groups. Ask yourself: Is there an adversarial relationship between security and the rest of the technology organization? Are they seen as watchdogs or partners?

Most likely, security is not an integrated part of the delivery value chain. Security tests and audits are seen as hurdles and barriers to continuous delivery. Address these organizational and process issues first. Make the development team responsible for building a secure environment and provide them with self-service tools so that they can test the software. Make the ops team responsible for running a secure environment. Give them the tools and metrics to be active partners on the watchtower.

 Troy Shafer

@shafertech

Troy Shafer is the owner of Shafer Technology Solutions, Inc. in Knoxville, TN. He has worked in IT for 20 years from service desk, system admin, network engineer, developer, Director of IT, and now IT Consultant.

“Threat modeling needs to be high on the list and early in the process…”

It provides the road map of how to proceed with implementing SAST (static application security test) and DAST (dynamic application security test). Not only will Threat Modeling expose vulnerabilities in your architecture and design, but it will also get developers in the mindset of an attacker. SAST and DAST will incorporate information from threat modeling as well as other tests that fit the project goals. When developers and project leaders are more aware of their risks, they will approach the project with a higher security awareness.

 Michael Morris

@Topcoder

Michael Morris is the CEO of Topcoder (a global community of 1.4M+ design, development, and data science experts disrupting enterprise software innovation through competition) and Global Head of Crowdsourcing for IT services leader Wipro. A gig economy expert, he speaks worldwide about cultivating a passionate workforce to drive the transformative nature of digital asset development.

“It has been about ten years since DevOps has permeated IT’s vernacular…”

When done well, DevOps can shorten the software development lifecycle, break down developer/business team silos, and target critical application features (e.g., performance, security, etc.). Within any successful DevOps strategy, IT security must be integrated from end-to-end (hence, the rise of the term DevSecOps, which requires thinking about application and infrastructure security from the very start of your DevOps initiative). Whichever term you use, DevOps or DevSecOps can fail if outdated security practices are used.

A key consideration in building DevSecOps is to adapt the right mindset in the approach. Succeeding in a DevSecOps initiative is about ensuring security without compromising innovation.

One of the biggest myths surrounding DevSecOps is that it’s a cloud service, or set of tools, you can just “buy off the shelf.” In reality, DevSecOps is a culture — along with tools, processes, and organizational structure — that facilitates getting features from developers to customers in the most secure way possible. Thinking that DevSecOps is simply something that you purchase will never get you where you need to go. Although you certainly “invest” in DevSecOps (via tools, people, etc.), success can’t simply be bought and uploaded for use. DevSecOps can fuel innovation and productivity, as well as speed time to market, when people/tools/processes seamlessly fit together.

A word of caution: Once DevSecOps is in place, you’re still not done. Just as you never stop finding new ways to deliver value to, or innovate for customers, DevSecOps is a living, breathing entity. Barriers must constantly be broken down, relationships must be built and rebuilt, and processes and tools need to be reviewed for effectiveness to ensure scalability and security.

 Frank Koehl

@newcontext

Frank Koehl has 20 years’ experience designing, building, and delivering software products. He founded 2 companies and launched 12 products in his work across the healthcare, technical, and environmental sectors. He now advises companies on their technical and business processes through New Context, a leader in DevSecOps services.

“Consensus has formed around some basic-yet-critical ideas…”

Pipelines ultimately mirror company culture, because interests within an organization apply their influence to the process. It’s therefore imperative that all stakeholders buy into the vision, and all participants are empowered to make changes that further the vision.

Keep job processes as simple as possible. Fewer moving parts means less chance for breakage, a narrower security footprint, and faster performance. Plus, if your pipeline is too complicated or slow, individuals may attempt workarounds to circumvent the system, leading to security risks and potential failures.

Don’t get paralyzed overthinking it. Just start doing it and refine as you go. Side benefit: A simple start leads to simple job processes.

 Steve McDonnell

@PrepaidFS

Steve McDonnell is the Senior Information Security Manager at PFS. Steve is an accomplished Information Security leader with a wealth of experience in highly regulated entities. Today, he is empowering FinTech employees with the latest security training they require to successfully carry out their work in four offices in three countries.

“There are many reasons why DevSecOps is important…”

It covers several areas such as Dev Code Analysis, Code Change Management, Vulnerability Management and Mitigation, Threat Investigations, Continuous Testing, On-Going Training, and wider Compliance Monitoring, such as PCI DSS/GDPR. However, probably the most important consideration when building a DevSecOps environment is to ensure you have the correct sponsorship at board level to ensure the model being implemented is understood, sufficiently resourced, and ultimately successful. The strategic decision on whether the service is cloud-based/selecting the right cloud service provider is also key.

 Jarrod Overson

@jsoverson

Jarrod Overson is the head of engineering at Shape Security. He’s a noted security expert, a sought-after speaker, and an O’Reilly Author.

“The most important part of a DevSecOps pipeline is…”

Ensuring that builds are broken at the earliest possible integration point. This does rely on a good CI/CD process that is already in place. Developers should check work into branches, and CI should already run developer and QA integration tests against those branches automatically. DevSecOps teams need to have an extensible spot to add additional checks during this process to catch issues early in development.

Applying security checks to feature development adds cost by default. Any cost-increasing check needs to be accounted for as early as possible. Otherwise expectations across teams and stakeholders become misaligned. When a feature looks complete and has been messaged as complete but is being held up by DevSecOps, then it’s too late. Security, QA, Accessibility, and other concerns are often the first to be cut when they are the sole blockers to a release.

 Tom Smith

@DZoneInc

Tom Smith is a research analyst for DZone.com.

“The most important consideration in building a DevSecOps pipeline is…”

Integrating automated security testing from initial code to production and beyond.

If automated security is not automated into the DevOps pipeline, you will not be able to make code secure on the backend, and you will not be able to deliver quality, secure code in a timely manner — the promise of DevOps.

 Kaushik Mysur

@Instaclustr

Kaushik Mysur is the Director of Product Management at Instaclustr.

“A traditional security framework where security testing and enforcement is done just before a release essentially negates every benefit that DevOps offers…”

Business agility, time to market, and an empowered development team. With DevSecOps, you bake security processes and tools into the development and CI/CD pipeline, you focus on building features securely rather than standalone security features, security is owned by every team in a release pipeline (not just the dedicated security team), and essentially, there is a shift in mindset from Reactive security (a friction to business value) to Proactive Security — a business value enabler.

The focus for the security team in this new paradigm will be to engage, collaborate, and enable development teams with required tools and standards to deliver security iteratively along with the rest of the product features. They should also work with the DevOps team to build security automation into CI/CD — things like static and dynamic code analysis, security testing, key management, checks on configuration, RBAC, logging and audit, etc. The ultimate goal should be to build security enforcement that operates at the speed of DevOps. With the growing adoption of microservices architecture and containerized apps, which has seen continuing headlines around breaches in containerized apps on Kubernetes, adopting DevSecOps is even more critical as it enables security to be built into each loosely coupled moving part of a product running on pods.

 Andrei Lipnitski

@ScienceSoft

Andrei Lipnitski is an ICT Department Manager at ScienceSoft, an IT consulting and software development company. He has 8 years of experience in managed IT services. Andrei also has an extensive background in solution architecture design and deployment, including CRM and data management solutions deployment in the cluster environment.

“To make your DevSecOps pipeline succeed, you must ensure strong collaboration between…”

Development, testing, operations, and security teams, and automate as many security processes as possible. Collaboration allows combining all the teams’ efforts and leveraging their strengths to achieve business goals faster. Automation, in its turn, significantly increases the speed of performing routine security activities like software vulnerability scanning, compliance testing, etc.