Blog Categories Application Security Cloud Security Compliance Container Security & Orchestration DevSecOps General Professional Development SOC Threat Intel Threat Stack Uncategorized Subscribe Now x Subscribe to Our Blog! DevSecOps 2 Min Read Trash Taxi: Taking Out the Garbage in Your Infrastructure Pat Cable August 29, 2019 Trash Taxi: A Lifecycle Management Tool for Superuser Discovery & Cleanup One of the security challenges we had at Threat Stack was managing developer access to production infrastructure. We already have a set of controls around managing scoped access depending on role (and if you’re on-call), and we have a provisioning system that uses hardware-backed keys for all access, which is great as well. An edge case around all of this is the ability for our developers to run arbitrary privileged commands in Production. At face value, this is scary! What if an insider leaves something behind? What if they cause more damage to the infrastructure in the process of running arbitrary privileged commands? But the reality is: When your infrastructure is processing tens of billions of events per day, you are absolutely going to find debugging issues that become impossible to debug in a development or QA environment. How can we ensure that engineers can run commands in Production while maintaining the cleanliness of our infrastructure? At Threat Stack, our Platform Security team’s job is to figure out solutions to problems like this one. Our solution to this was developing a tool called Trash Taxi. Users who would potentially run a command like sudo -i or sudo bash are no longer allowed to run that command; instead they run sudo nt. Running this command will register the server as having had the “seal broken,” and on the next Trash Taxi pickup, the machine will be terminated. There are a few safety controls on this kind of collection — the end user can specify certain “trash holidays” either by an EC2 Role or Type tag that you may use in your infrastructure. This allows you to track when your developers may need to execute commands on a sensitive database host — so you can go and have a conversation about why that may have needed to happen (or not). Some infrastructures will have hosts that are challenging to terminate — that’s okay! You can use this data to justify spending engineering time in the right places to improve those processes if necessary. Trash Taxi Resources Trash Taxi was released at Black Hat Arsenal this year. If you’re interested in hearing more about it, I did an interview with Dark Reading that you can listen to: More information, including documentation and configuration information, is available at https://trash.taxi. Interested in Taking Trash Taxi for a Test Drive? Take a look at the Getting Started guide. Make your Configuration file. Use the garbaged API to pick up the trash! Tags:Trash TaxiTrash Taxi Lifecycle Management Tool You Might Also Like... Balancing Security and Your On-Call Rotation Using Deputize Authkeys: Making Key-Based LDAP Authentication Faster Three Homegrown SecOps Tools Used by the Threat Stack Team Private: Black Hat 2019 Recap: Transformation & the New Cybersecurity Culture About Pat Cable Patrick Cable is Director of Platform Security at Threat Stack. As an infrastructure security engineer, Patrick focuses on ensuring the security of the Threat Stack Platform by collaborating with other departments, implementing security tools, and building new technology to make security easier for everyone in the organization. Prior to working at Threat Stack he was Associate Staff in the Secure and Resilient Systems Group at MIT Lincoln Laboratory where he worked on improving cloud security in research environments. View more posts by Pat Cable Request a Demo Share this Blog