How We Integrated Rust Into Threat Stack’s Operations Workflow


Note: The following post is related to Sensu, a monitoring tool for internal infrastructure health and alerting. If you use Sensu (https://sensuapp.org/) for internal monitoring of your own infrastructure health, this could be useful for you. However, this tool does not integrate with Threat Stack services and is not intended or supported for any such use case. It is a tool that we use internally, and we have released this with the intention that it may be helpful to the wider open source community.


Tooling is an integral part of operations at Threat Stack. On the Operations team, our job is to enable both ourselves and the Development team to work more effectively. When I started at Threat Stack almost a year ago, my role primarily centered on improving our tooling to create more granular control over our environment. My first project was creating “shush,” an operations tool for temporarily silencing monitoring checks in Sensu during maintenance. Up to that point, we had had less granularity in our check silencing capabilities for routine maintenance. While we could silence groups of checks and checks coming from a particular node, we were not able to silence single checks or a subset of checks on these hosts. After we discussed the requirements for this tool, I ultimately suggested that it be written in Rust.

In this post I describe our experience integrating Rust and also cover the benefits of using Rust in an operations workflow both technically and from a human factors perspective. Read more “How We Integrated Rust Into Threat Stack’s Operations Workflow”

VPNNotify: A VPN Notification bot for Slack

In an earlier post, we talked about how we implemented centralized authentication at Threat Stack. This project initially allowed us to create clearer access control for our servers. A side benefit of this work has allowed us to write tooling around common authentication processes.

One thing we’ve wanted to do is create an alert when folks are using a VPN to connect to one of our environments. In the event of a stolen laptop and stolen credentials, a user could be alerted to someone logging in with their credentials. With OpenVPN, performing actions on a client connect is possible using a client-connect script, so in the tradition of writing small Go applications to improve visibility, we did just that.

For the last few months our Slack bot VPN Notifier has been letting our engineers know when they connect into a Threat Stack environment. We’ve now done the work to open source the tool so that others can use and improve on it. We specifically mention improve, because our tool has limitations: The current version does extremely basic environment checking, and extremely basic alert suppression. Our hope is that we can collaborate with others who want to take this tool the extra mile. Read more “VPNNotify: A VPN Notification bot for Slack”

Authkeys: Making Key-Based LDAP Authentication Faster

Authkeys, Threat Stack’s new open source tool, performs LDAP lookups of SSH keys without the need for using scripts or other interpreted code.

You may recall from an earlier post that we’ve set up centralized authentication here at Threat Stack. Our motivation for doing so centered on the desire to achieve clearer access control for the servers that power our platform. By doing this, we no longer need to use Chef to deploy the majority of users to servers. Rather, we can use an internal application to add, lock, and update users and their associated metadata.

Read more “Authkeys: Making Key-Based LDAP Authentication Faster”

Balancing Security and Your On-Call Rotation Using Deputize

Threat Stack, like many other Software-as-a-Service providers, has an on-call rotation. During any week, two members of our engineering organization are tasked with responding to alerts across the platform they build and maintain. These two engineers are also responsible for a myriad of other services as well that provide support to the infrastructure: services that provide metrics and monitoring, log capture and collection, authentication, etc.

This presents a security issue with regard to access control: should all staff have access to all servers all the time? In early start-up life this is unavoidable. But as an organization matures and grows, it becomes a bigger risk. Administrator and similarly scoped credential theft is a goldmine for attackers, so we wanted to improve our story around internal access control.

Unwrapping who needs access to what is always an evolving task, but we put in the work to figure out who goes where and why, and then created groups to control that access. Since we already use groups as a way to control who can log into specific machines, and we use PagerDuty to assign on-call rotations, it seemed like we could create a tool that would query PagerDuty and update our on-call group. So we did! And as a gift to you, we’ve open sourced it.

Read more “Balancing Security and Your On-Call Rotation Using Deputize”

Reliable UNIX Log Collection in the Cloud

One way organizations can improve their security and operational ability is to collect logs in a central location. Centralized logging allows engineers across the entire organization to have a “common view” of the system under load, and can provide vital shared context when things go wrong.

Over the last few months, we at Threat Stack have been reworking how we handle all aspects of our logging system. This project encompasses everything, from the content of our log data to the infrastructure that collects it. In this post you’ll learn about how our internal applications send log data, where they send it to, and the trade offs we considered in making our collection system reliable. Read more “Reliable UNIX Log Collection in the Cloud”

Parsing Simple Grammars in Scala With parboiled2

parboiled2 is a Macro-Based PEG Parser Generator written in Scala. It has become our preferred tool for creating parsers for simple grammars. It offers a fairly simple syntax for creating parsers and boasts significantly better performance over Scala Combinators.

To illustrate its use, we will use this post to build a parser for a simplified version of the Slack Assisted Search. Read more “Parsing Simple Grammars in Scala With parboiled2”

Test Systems: The Soft Underbelly of System Security

Test systems are the guts of your overall system design. Test systems embody an incredible amount of the history of how your team’s code and development practices turn into the tools, applications, and services you provide to your users. Most importantly, these systems show how your systems, tools, and applications mature and refine, and every brilliant and embarrassing decision your team made to create your product can often be found there.

But the contents of test environments, whether it’s test frameworks, CSV files, or a rainbow of test data, can be very sensitive, and more importantly, your test systems could contain a decade or more of information related to the development of your systems, tools, and applications. In spite of the potential sensitivity of the data in these systems, the often ad hoc, or ShadowOPS, nature of their development, frequently means that test systems do not undergo the same security scrutiny as other parts of your development environment or production systems, and this means that the risk associated with them probably outstrips the regular investment that has been made to secure them. So from a security point of view, it’s time to right the balance — to toughen up the soft underbelly of your test systems so they don’t undermine your test, dev, and production environments. Read more “Test Systems: The Soft Underbelly of System Security”

Securing User Credentials With the YubiKey 4

I’m a big fan of the YubiKey 4.

The YubiKey is a security device that originally outputted a 44-character “one time password” that could be decoded and mathematically verified and used as a second factor for authentication. Over the last few years, improvements to the devices mean that they can also perform other important functions, such as storing:

  • Identity, Signature, and Encryption Certificates
  • U2F data for websites (GitHub and GMail, among others, support this)
  • GPG Keys

If you’re looking to set this up on your own, read on to learn how this extra functionality helps your security game, and how you can configure services to use it. Read more “Securing User Credentials With the YubiKey 4”

Unit Testing With Webpack & Mocha

After moving our build infrastructure to webpack, one of the hurdles we had to overcome was finding a good way to run unit tests. Quite a few tutorials and how-tos are available for using mocha with webpack, but none of them gave us all the things we wanted from our test setup.

Therefore, as described below, we defined our requirements for the testing infrastructure and created a setup that enables us to build a test bundle from any number of spec files in a directory, incrementally rebuild on each spec change, and rerun the test suite via mocha on every change. Read more “Unit Testing With Webpack & Mocha”