Balancing Security and Your On-Call Rotation Using Deputize

Threat Stack, like many other Software-as-a-Service providers, has an on-call rotation. During any week, two members of our engineering organization are tasked with responding to alerts across the platform they build and maintain. These two engineers are also responsible for a myriad of other services as well that provide support to the infrastructure: services that provide metrics and monitoring, log capture and collection, authentication, etc.

This presents a security issue with regard to access control: should all staff have access to all servers all the time? In early start-up life this is unavoidable. But as an organization matures and grows, it becomes a bigger risk. Administrator and similarly scoped credential theft is a goldmine for attackers, so we wanted to improve our story around internal access control.

Unwrapping who needs access to what is always an evolving task, but we put in the work to figure out who goes where and why, and then created groups to control that access. Since we already use groups as a way to control who can log into specific machines, and we use PagerDuty to assign on-call rotations, it seemed like we could create a tool that would query PagerDuty and update our on-call group. So we did! And as a gift to you, we’ve open sourced it.

Deputize is the tool that we created and use to manage our on-call rotation. Deputize uses PagerDuty’s API to get the email addresses of the engineers on call, then compares that to the named users in the on-call group, and updates accordingly. When changes occur, we’ve added in integration to notify Slack and log to a tool like Graylog as well. This allows us to ensure that everyone who needs to access all hosts can (in an automated fashion), while keeping that access limited to the time they’re on-call.

Using Deputize

Deputize has a few prerequisites:

  1. Deputize builds on PagerDuty, so you’ll need to be using PagerDuty to manage your on-call rotation.
  2. We use LDAP to manage authentication and group access.
  3. Secrets such as the Slack API key, the PagerDuty API key, and the password of the LDAP user authorized to make changes to the on-call group are stored in Vault.

We provide information on how to configure Deputize in the README.md file located with the Deputize source, but the summary of it is that you’ll provide a path to a JSON config file via an environment variable. You can run Deputize on a schedule in a variety of ways — we have Sensu run Deputize like a “check” on an interval, but you could also run the tool via cron as well. Then, sit back and watch Slack for your on-call schedule to change automatically.

Wrapping up

Here at Threat Stack, we’re always looking for ways to improve our internal security story. We’ve been running Deputize since early January, and it has faithfully kept our on-call group updated (including overrides), which is a nice process to have automation around. We’d love to hear if Deputize works for you too — and we look forward to seeing future comments and pull requests!

Interested in Deputize? Check it out on GitHub.