At Threat Stack, we use our own intrusion detection platform to protect Threat Stack. This gives us critical visibility into security events and alerts tied to our AWS infrastructure and instances, an all too popular target. But our infrastructure extends beyond AWS into additional vendor-managed solutions such as Cloudflare, SalesForce, corporate email, and others. So a key question is: How can we not only monitor those platforms, but also use the data from these logs to drive security priorities?
With that in mind, we set out to create a new custom internal app that can receive, store, and perform actions on information from all of these different sources. We opted to build this internal pipeline (some would call this security orchestration) instead of buying an off-the-shelf product because our security team indexes so highly on engineering and programming. We felt we could take an event-driven framework in a language we all knew and easily extend it to meet our needs, incorporating our internal detection and automated response frameworks, a choice we would not have made if our team or organization looked different.
Contextual Understanding From a Mountain of Data
What would this application take in? Our initial thought was that we could enrich Threat Stack’s alerts with additional information about our environment. We configured the Threat Stack webhook integration to send all alerts to this orchestration application, which then leveraged Threat Stack’s own robust API endpoints to enriched the data, output the data to Slack or PagerDuty, and then store it for longer-term archiving. Soon after, we started adding in CloudFlare log capture information because this provided data for some traffic analysis.
Where is that data stored? Enter Graylog, an open source log aggregation, indexing, alerting, and data visualization tool that our Operations team had already deployed for centralized application logging. All of the log, event, and alert data from those heterogeneous security data sources is sent there. Since Graylog was already set up internally, security got the value of Operations’ investment without having to invest too heavily ourselves.
The orchestration application we built also tags alerts with an identifier so we can analyze alerts over time. We dislike thinking of alerts as just “things you may or may not want to respond to.” Instead we want to see them as valuable analytical data that can drive our security priorities like a product team uses web analytics to define and measure feature work.
How Has This Helped Us?
To get insight into any potential data leaks, we added a rule to look for relevant log events, and if any are found, Graylog sends the team an immediate alert via Slack so they can investigate and possibly schedule follow-up development work to remediate. We also defined a custom query to find all alerts, including those from Threat Stack, for the past day, week, and month to understand how our internal users’ and applications’ behavior is changing over time as we implement our project work. Using the Quick Values features in Graylog on those results, we added pie charts to a custom-configured dashboard so we can see the top five most frequent types of alerts. This gives us a baseline of data, and through visiting this dashboard on a regular cadence, we can see whether our platform is trending more or less risky over time.
In one particular instance, we needed to investigate some request traffic into our platform from a certain IP address range to see whether any abnormal patterns existed. Since we were replicating our Cloudflare logs to Graylog through this orchestration app, all we had to do was create a query for requests coming from those IP addresses for the time frame we were interested in. Showing the results in a graphical manner with pie charts and histograms gave us quick insight into the patterns.
Wrapping Up . . .
Feeding new data into Graylog with our orchestration application has proven to be an important and value-add ability, giving us visibility and analytical data to drive our platform security program. We look forward to thoughtfully pulling in more data, using our APIs to contextually enhance the data, continuing to integrate it further with our internal automation and event processing layer for more complex business logic. Our orchestration application can do a whole lot more — so we’re looking forward to sharing some of those practices with you in the future.