Live Demo
Blog   >   Cloud Security   >   Security Rules with Anomaly Detection: Capture the Known and the Unknown

Security Rules with Anomaly Detection: Capture the Known and the Unknown

Any cybersecurity team that has had to respond to a compliance audit knows that it’s a lot easier to monitor for specific risks as they occur, rather than gathering logs and combing through data after the fact. And any organization that has suffered a cloud security breach knows there’s always some behavioral anomaly that, if noticed sooner, could have prevented the incident altogether. Continuously monitoring for known risks, while also surfacing unknown anomalies as soon as possible solves this problem.

Unfortunately, many cloud security tools force you to consider these capabilities as tradeoffs. But what if you didn’t have to choose? In this blog we’ll discuss why it’s important to be able to detect and respond to both types of problems.

Monitor the known

For compliance, it’s best to collect evidence in real time. Similarly, producing compliance artifacts can largely be accomplished with visibility into your existing security controls. As a result, you’re likely already accounting for many, if not all, of the control points that auditors will assess.

For example:

  • Who has accessed sensitive data?
  • Which files have they touched?
  • Who can log into production systems?
  • What commands did they run once they logged in?

These problems map well to alerting rules. While rules can be used for many use cases, security and compliance alike, we see advancing compliance audits as a predominant one. Additionally, rules align with areas that auditors typically focus on during their evaluations.

For rules to work, they must run continuously, so you don’t miss any of these behaviors. You also need long-term data retention, so you can properly scope evidence. Monitoring for known behaviors this way will speed up the security audit process, saving your team’s time and registering fewer billable hours with the accountants.

Threat Stack’s rules engine is production-proven, with more than six years of experience building out an extensive set of alerting rules to define risks that are relevant, known, and real. We monitor systems continuously across the full cloud stack, and retain alert data for a full year. And for advanced reporting and analysis, you can export all of your enriched Threat Stack telemetry out of our platform and into your own data lake. Threat Stack customers use all of these features to identify and respond to known security risks and satisfy audit requirements faster.

Monitor the unknown

Anomaly detection provides coverage and context for risky activities and behavior patterns that would be difficult, or impossible, to predict in advance and account for with rules.

Machine learning models can accelerate anomaly detection, but you can’t evolve past basic statistics without first investing in deep telemetry collection and extensive alerting rules. Together, these sources contribute the essential data needed to train advanced ML models. Likewise, to increase the relevance of their findings, intent data related to how security teams are using rules must inform the ML algorithms.

Basic ML models can still provide helpful coverage, but their relevance will vary. For example: say someone opened an unusual open port in your environment — maybe a harmless developer experiment, or part of a larger attack. Either way, the difference between 500 instances of port 22 and 1 instance of port 6379 is clear, but without more context it’s hard to determine its relevance. (What happened next? If an attacker uses common commands that conform to frequently observed patterns, they will be harder to spot as anomalies.)

In this way, products that operate with basic ML models become black boxes. You don’t have insight into the raw data they’re using to calculate statistics, and telemetry is incomplete — likely diminishing investigation and threat-hunting efforts.

Advanced ML models can provide more relevant findings. For example: An attacker could use a command that operators use frequently in your environment, but combine it with obscure parameters and options you hadn’t known to capture in your rules. A basic ML model that uses less data might not flag this anomaly because it commonly sees this command. But advanced machine learning techniques that are trained with comprehensive telemetry will capture the string variance in the additional options and parameters, detecting an anomaly that might have otherwise gone unnoticed and could indicate a significant security incident or widespread breach. Plus, you’ll have all the metadata needed to account for this edge case in an alerting rule.

Here at Threat Stack we are unifying advanced ML and rules, allowing our customers to benefit from both approaches. The rules feed the ML, and the ML feeds the rules. Compliance is covered, and anomaly detections are relevant — which means audits will run more smoothly, and you’ll decrease mean time to know (MTTK) for issues that could negatively impact your cloud security security posture.

Complete coverage across the known and unknown — coming soon

Threat Stack will soon feature multiple detection methods to help customers adapt to a wider range of cloud security monitoring use-cases. You’ll no longer have to choose between rules and ML.

Stay tuned for more from Threat Stack on how you can gain full visibility across known threats and the previously unknown factors affecting your risk posture. Until then, check out our recent webcast with SANS, “More than a Buzzword: How to Deliver on the Promise of Machine Learning,” for a look at how you can leverage advanced ML to accelerate your cloud security and compliance initiatives.