Trash Taxi: A Lifecycle Management Tool for
Superuser Discovery & Cleanup
One of the security challenges we had at Threat Stack was managing developer access to production infrastructure. We already have a set of controls around managing scoped access depending on role (and if you’re on-call), and we have a provisioning system that uses hardware-backed keys for all access, which is great as well.
An edge case around all of this is the ability for our developers to run arbitrary privileged commands in Production. At face value, this is scary! What if an insider leaves something behind? What if they cause more damage to the infrastructure in the process of running arbitrary privileged commands? But the reality is: When your infrastructure is processing tens of billions of events per day, you are absolutely going to find debugging issues that become impossible to debug in a development or QA environment. How can we ensure that engineers can run commands in Production while maintaining the cleanliness of our infrastructure?
At Threat Stack, our Platform Security team’s job is to figure out solutions to problems like this one. Our solution to this was developing a tool called Trash Taxi. Users who would potentially run a command like sudo -i or sudo bash are no longer allowed to run that command; instead they run sudo nt. Running this command will register the server as having had the “seal broken,” and on the next Trash Taxi pickup, the machine will be terminated.
There are a few safety controls on this kind of collection — the end user can specify certain “trash holidays” either by an EC2 Role or Type tag that you may use in your infrastructure. This allows you to track when your developers may need to execute commands on a sensitive database host — so you can go and have a conversation about why that may have needed to happen (or not). Some infrastructures will have hosts that are challenging to terminate — that’s okay! You can use this data to justify spending engineering time in the right places to improve those processes if necessary.
Trash Taxi Resources
Trash Taxi was released at Black Hat Arsenal this year. If you’re interested in hearing more about it, I did an interview with Dark Reading that you can listen to:
More information, including documentation and configuration information, is available at https://trash.taxi.
Interested in Taking Trash Taxi for a Test Drive?
- Take a look at the Getting Started guide.
- Make your Configuration file.
- Use the garbaged API to pick up the trash!
