Threat Stack Adds Advanced Supervised Learning to ThreatML™ for Detection-In-Depth

John Pinkham

Published June 07, 2022

Threat Stack is now F5 Distributed Cloud App Infrastructure Protection (AIP). Start using Distributed Cloud AIP with your team today.

Because Anomaly Detection Is No Longer Enough for Cloud-Native Security

Until now, organizations securing cloud-native infrastructure had to rely on anomaly detection. Even the promise of this type of machine learning was constrained by technical difficulties and lack of data, preventing threat detection-in-depth.

No longer.

Threat Stack’s latest version of ThreatML is now powered by supervised machine learning. Available to Threat Stack customers, ThreatML now moves beyond just anomaly detection that is the current industry standard. ThreatML delivers tightly-focused, high-efficacy threat detection based on behaviors for a detection-in-depth approach that combines Threat Stack’s sophisticated ruleset with supervised machine learning.

Moving Beyond the Current State of Intrusion and Threat Detection

DevSecOps teams and other security groups are continually constrained in properly running security operations. According to a recent study of over 200 DevSecOps team directors and managers, CISOs (Chief Information Security Officer), cloud security engineers and architects, and others, cloud security teams routinely face:

Staffing issues / lack of manpower
Small budgets
Perception of “no-value-added” from C-level leaders
Time and resource demands
Too many competing daily priorities
Pressure to achieve operational efficiency

In addition, more and more sophisticated threats are creating evolving threats and vulnerabilities that security teams need to stay on top of.

Most cloud security vendors look to offer a combination of cloud security and operational efficiency, so they attempt to solve these issues by offering anomaly detection and reporting. That is, they develop programs and solutions that focus on finding and reporting on items which appear to be different than what has historically been an organization’s baseline behavior.

Why? Quite simple: Tools and solutions that only surface anomalies, or what is different from normal baseline behavior, do not require much tuning, training, triaging, or reviewing of alerts. This gives customers a necessary intrusion detection method that alleviates pressure on running daily security operations. In fact, some companies boast of providing “only a couple” of anomaly detection reports a day. As we’ve written about in the past, having an artificial limit on the number of generated alerts is not a good metric – and in fact can be dangerous to an organization’s cloud-native security. With anomaly detection, organizations must always ask themselves: Which threats and intrusion alerts can we afford to miss?

Having just one method of detection like this, by itself, is insufficient.

There are a couple of reasons anomaly detection is not enough to secure cloud environments:

Abnormal, anomalous, or outlier behavior from a typical baseline is not always a threat. Alerting on this type of behavior can create a false positive.
Behavior that appears normal doesn’t necessarily mean it is good. Ignoring certain behaviors just because the rules consider them normal generates false negatives.

Tools that only offer one detection method like anomaly detection miss out on critical behaviors that indicate real threats. These systems are designed to only surface what looks different. In short, using anomaly detection on its own sacrifices security for the sake of operational efficiency.

How Threat Stack Expanded ThreatML Based On Customer Feedback

As the engineering teams at Threat Stack spoke with customers, it became increasingly obvious: DevSecOps and other cloud security teams need a robust, innovative cloud security tool that provides several solutions. It must:

Provide comprehensive coverage of both known and unknown security threats;
Eliminate false negatives;
Recognize and deal with false positives
Keep operational constraints low
Limit findings to real, actionable threats
Create filters and models that do not miss critical behaviors
Be easy to deploy, manage, and run daily

Moving Beyond Just Anomaly Detection to Threat Detection-In-Depth

To meet these customer needs, Threat Stack looked to create a detection-in-depth approach that could uncover both known and unknown threats, while eliminating the false negatives. The goal was to advance past anomaly detection and provide a better picture of a customer’s environment.

The solution? The advanced version of ThreatML, which uses supervised learning to deliver high-efficacy threat detection on behaviors through a detection-in-depth approach. Threat Stack’s novel use of supervised learning for cloud security allows security teams to keep their organization’s data secure while providing operational efficiency.

ThreatML with Supervised Learning: Machine Learning Done Right

A main reason vendors do not leverage supervised learning is that using it requires labeling billions of events daily to train the algorithms. In other words, supervised learning needs data, a lot of data, and that data must be classified. And classifying data can be an extremely labor-intensive activity, requiring many data engineers.

ThreatML takes a novel approach to supervised learning by using Threat Stack’s extensive rules engine to classify and label more than 60 billion pieces of data a day, in real time. This type of labeled data at scale is a requirement to fully realize the potential of supervised learning.

Once a behavior passes through the rules engine, it can be analyzed. Threat Stack created an inference engine which uses the labeled and classified data to make predictions about behavior. The inference engine determines if the behavior is predictable based on data from surrounding events. Behavior that is unpredictable represents a high-priority threat, which gets surfaced to the customer as an alert.

Adding supervised learning to our rules engine gives Threat Stack customers multiple detection methods to catch threats to their cloud environment. It allows organizations to answer the question: “Given the historical behavior in this workload, was this behavior predictable or not?” Predictable behaviors can be safely ignored, while unpredictable behaviors represent real, actionable threats.

As a result, ThreatML allows customers to focus only on the highest priority threats to their environment. This limits the alert fatigue and uses less resources. The supervised learning approach relies on rules to train models automatically and continually, giving customers a low-touch way to get high-efficacy threat detection. It is similar to the operational efficiency anomaly detection promises – but the supervised learning method does not risk missing the highest priority threats to an organization’s environment.

To discuss how Threat Stack’s new ThreatML with supervised learning can help your organization’s daily cloud security operations, contact us today.