Upcoming Webinar — Good, Fast, or Secure? Why DevOps Means You Don’t Have to Choose

Live Tuesday, March 27 at 1:00 p.m. EST

Click here to register.

Overview

Common wisdom holds that, when it comes to software releases, you can only have two of: good, fast, or secure. But we don’t agree at all. When DevOps is implemented thoughtfully and holistically — and when security is brought into the process early — it’s entirely possible to release high-quality, secure code as quickly as the market demands.

In this webinar, we’ll walk you through exactly how Threat Stack has avoided sacrificing security on the altar of speed and share best practices to help you achieve the holy trinity of good, fast, secure code at your organization. Read more “Upcoming Webinar — Good, Fast, or Secure? Why DevOps Means You Don’t Have to Choose”

How Threat Stack Does DevOps — Series Overview

Pete Cheslock, Threat Stack’s Senior Director of Operations, has just published a four-part blog series that gives deep insights into his experience “doing DevOps” at a variety of companies — in particular, his highly successful experience building DevOps practices into the fabric of Threat Stack virtually from day one.

We encourage you to read the entire series: It’s loaded with great accounts of what works and doesn’t work in real-life environments  — there’s nothing academic about Pete’s approach — and also offers up lots of practical advice you can draw on if you’re trying to figure out the best way to implement DevOps in your organization. But before you dive in, we thought we’d offer up a reader’s digest version to get you going. Read more “How Threat Stack Does DevOps — Series Overview”

How Threat Stack Does DevOps (Part IV): Making Engineers Accountable

Early on at Threat Stack, we focused on giving engineers the tools and ownership over their applications that would empower them to deploy and manage their applications in a safe way without causing customer downtime or other issues. As a small, but rapidly growing company, this is necessary for survival. For most of the last four years, Threat Stack has only had a two- to three-person operations team. With a such a small team, we understand that we can’t have our hands on everything that happens in production. It just doesn’t scale, especially given how difficult it can be to hire engineers is this competitive market.

In this post, we’ll take a look at how you can better scale your organization by employing the DevOps best practice of giving engineers fundamental responsibility for their code. Read more “How Threat Stack Does DevOps (Part IV): Making Engineers Accountable”

How to Integrate Security Into a DevOps World

Introduction

by Pete Cheslock, Senior Director Operations, Threat Stack

Today we’re pleased to have Franklin Mosley, Senior Application Security Engineer at PagerDuty, contribute to our blog.

Drawing on his extensive experience as an information security professional, Franklin takes a detailed look at the how’s and why’s of integrating security into a DevOps environment, and provides great tips on how you can start making the transition to a DevOps culture at your organization.


I have been in security for many years, so I have heard many of my colleagues complain that developers and operations have little regard for security. But my perspective is a little different: I used to be a software engineer, so I understand the challenges faced in getting software developed and deployed. To that end, I want to share some of my experiences in this post, and hopefully pass along some valuable tips on how to effectively integrate security into your DevOps world. Read more “How to Integrate Security Into a DevOps World”

How Threat Stack Does DevOps (Part III): Measuring and Optimizing System Health

One of the most important things that any company can do to benefit from DevOps is define and implement useful, actionable metrics for visibility into business operations.

This is already standard practice in most areas of the average organization. KPIs drive sales and marketing teams, finance groups, and even HR. Yet, at many companies, having metrics for the application that brings in the money is an afterthought — or is not prioritized at all.

In this post, we’ll take an in-depth look at why application and infrastructure metrics should be baked into your engineering organization as early as possible, how to do it, and what tools can enable your success around this key area of DevOps. Read more “How Threat Stack Does DevOps (Part III): Measuring and Optimizing System Health”

How Threat Stack Does DevOps (Part II): Engineering for Rapid Change

Many organizations struggle with how and when to deploy software. I’ve worked at some companies where we had a “deploy week.” This was at least a week (or sometimes even longer) that was completely devoted to deploying huge amounts of software. The changes were so large and complex that deploying them would cause massive amounts of pain and suffering. It took hours every night for a week to deploy them, and it was too difficult to test all the changes one by one. So engineering and operations teams — not to mention customers — had to deal with broken updates until we could fix each one.

Additionally, because of the sheer volume of changes being deployed, the code was difficult to test. Systems would break in unforeseen ways, which led to distractions for engineering teams that would get called in to fix the issues. Imagine losing your entire engineering organization for an entire week every time you push out new software and updates! If this happens once a month, every month, it gets unsustainable fast.

Because I’d experienced this pain firsthand, I wanted Threat Stack to be different when it came to how and when we deploy code. That’s why we worked hard to embed DevOps best practices in our organization from the very beginning, starting with engineering for rapid change. In this post, I’ll walk you through what this means and why it is essential to doing DevOps well. Read more “How Threat Stack Does DevOps (Part II): Engineering for Rapid Change”

How Threat Stack Does DevOps (Part I): Best Practices in the Wild

As Senior Director of Operations at Threat Stack, I am repeatedly asked one question by our customers: “How does Threat Stack ‘do’ DevOps?”

One of my long-time pet peeves has been the abuse of the term “DevOps.” You can be a DevOps engineer, you can be a Director of DevOps, you can buy DevOps tools. But when people ask me “How does Threat Stack ‘do’ DevOps?”, I imagine them saying “How do you run Technical Operations?” See, it’s my belief that people often struggle at implementing DevOps because they don’t understand the complexity of technical operations. By this I mean managing the complexity of cloud environments, distributed systems, open source and home-built applications — and engineering them all for uptime and availability for customers. This is the crux of what it means to do DevOps well. Read more “How Threat Stack Does DevOps (Part I): Best Practices in the Wild”

High Visibility Ahead: Building and Using Orchestration to Set Security Priorities

At Threat Stack, we use our own intrusion detection platform to protect Threat Stack. This gives us critical visibility into security events and alerts tied to our AWS infrastructure and instances, an all too popular target. But our infrastructure extends beyond AWS into additional vendor-managed solutions such as Cloudflare, SalesForce, corporate email, and others. So a key question is: How can we not only monitor those platforms, but also use the data from these logs to drive security priorities?

With that in mind, we set out to create a new custom internal app that can receive, store, and perform actions on information from all of these different sources. We opted to build this internal pipeline (some would call this security orchestration) instead of buying an off-the-shelf product because our security team indexes so highly on engineering and programming. We felt we could take an event-driven framework in a language we all knew and easily extend it to meet our needs, incorporating our internal detection and automated response frameworks, a choice we would not have made if our team or organization looked different. Read more “High Visibility Ahead: Building and Using Orchestration to Set Security Priorities”

How We Integrated Rust Into Threat Stack’s Operations Workflow


Note: The following post is related to Sensu, a monitoring tool for internal infrastructure health and alerting. If you use Sensu (https://sensuapp.org/) for internal monitoring of your own infrastructure health, this could be useful for you. However, this tool does not integrate with Threat Stack services and is not intended or supported for any such use case. It is a tool that we use internally, and we have released this with the intention that it may be helpful to the wider open source community.


Tooling is an integral part of operations at Threat Stack. On the Operations team, our job is to enable both ourselves and the Development team to work more effectively. When I started at Threat Stack almost a year ago, my role primarily centered on improving our tooling to create more granular control over our environment. My first project was creating “shush,” an operations tool for temporarily silencing monitoring checks in Sensu during maintenance. Up to that point, we had had less granularity in our check silencing capabilities for routine maintenance. While we could silence groups of checks and checks coming from a particular node, we were not able to silence single checks or a subset of checks on these hosts. After we discussed the requirements for this tool, I ultimately suggested that it be written in Rust.

In this post I describe our experience integrating Rust and also cover the benefits of using Rust in an operations workflow both technically and from a human factors perspective. Read more “How We Integrated Rust Into Threat Stack’s Operations Workflow”

VPNNotify: A VPN Notification bot for Slack

In an earlier post, we talked about how we implemented centralized authentication at Threat Stack. This project initially allowed us to create clearer access control for our servers. A side benefit of this work has allowed us to write tooling around common authentication processes.

One thing we’ve wanted to do is create an alert when folks are using a VPN to connect to one of our environments. In the event of a stolen laptop and stolen credentials, a user could be alerted to someone logging in with their credentials. With OpenVPN, performing actions on a client connect is possible using a client-connect script, so in the tradition of writing small Go applications to improve visibility, we did just that.

For the last few months our Slack bot VPN Notifier has been letting our engineers know when they connect into a Threat Stack environment. We’ve now done the work to open source the tool so that others can use and improve on it. We specifically mention improve, because our tool has limitations: The current version does extremely basic environment checking, and extremely basic alert suppression. Our hope is that we can collaborate with others who want to take this tool the extra mile. Read more “VPNNotify: A VPN Notification bot for Slack”